Conversation
✅ Deploy Preview for nextflow-docs canceled.
|
Two extension points in nf-commons, both scheme-keyed and both optional: - ObjectStoreReader: a flat, no-delimiter listing that returns each object's size and mtime, and a ranged read of a single byte window. The listing is O(N/1000) calls regardless of how deeply the tree nests, where the NIO walk costs one delimiter-based LIST per subdirectory; the ranged read fetches one window instead of the whole object. Both default to null -- not implemented -- so a caller falls back to the hierarchical walk and the full read. - AtomicLockProvider: create-if-absent of a marker object, answered by the store itself rather than by a read-then-write. Object storage has no directory-create to race on, so this is the only portable way for two processes on different machines to agree on which of them owns a prefix. ObjectMeta carries the (size, mtime) pair the listing returns. Neither SPI has an implementation in this commit, so nothing changes: a scheme with no provider takes exactly the path it takes today. Assisted-by: Claude Opus 5 (1M context) Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Each provider gets a flat listing with per-object (size, mtime), a ranged read,
and a conditional create. The conditional create is the part that has to be
exact, because it is what two processes on different machines rely on to agree:
- S3 ifNoneMatch("*") -> 412 when the object exists
- Azure setIfNoneMatch('*') -> matched on the error CODE, not the status, with
a table of the six responses that mean 'taken'
- GCS doesNotExist() -> 412 via ifGenerationMatch=0
None is a read-then-write, so none has a window between the check and the create.
The ranged reads bound both the request and the buffer where the SDK allows it:
on GCS limit() alone still allocates the 2 MiB default chunk locally, while
setChunkSize() alone leaves the request open to the end of the object -- either
one by itself leaves half the waste.
GsStorageOptions holds the shared GCS client so the two extensions resolve one
client per JVM rather than one each, and the session config lookup they both
need.
S3Client gains putObjectIfAbsent, which carries the same ACL/KMS/SSE settings
its sibling writes apply: a bucket policy of the 'deny unless encrypted' kind
would otherwise reject the create with a 403, which the caller cannot read as
'lost the race'.
Assisted-by: Claude Opus 5 (1M context)
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Two extension points in nextflow.processor, both with defaults that preserve today's behaviour exactly: - TaskHasherFactory creates the TaskHasher for a task, or abstains. With no extension registered, or all abstaining, TaskProcessor does `new TaskHasher(task)` as before. Core's TaskHasher is untouched. - TaskCacheStrategy decides how a task is resolved against the cache, through the TaskResolver primitives the processor implements. DefaultTaskCacheStrategy is master's checkCachedOrLaunchTask loop moved verbatim -- the tries fold, the lock, mkdirs, submitTask -- so a run with no extension takes the same path it takes today. checkCachedOrLaunchTask becomes the dispatch. Supporting changes: - CacheFactory may resolve its own work directory and write it back to the session, so Session.init creates the cache before reading workDir. Everything below -- the work dir creation, the observers, the WorkflowMetadata snapshot -- must see the effective value, or workflow.workDir names a directory no task uses. A throw after that point now closes the cache rather than leaking it. - CacheStore.updateEntry, a default method, so a store composed of several members can route an update back to the member that served the read. - CacheDB.dispatchWrite, an internal refactor so a failed async cache write is logged with its cause instead of being swallowed. - FileHelper.getTaskHashFromPath tolerates an optional `-N` suffix on the work directory leaf. nf-lineage resolves every task input to its producing task by parsing that path (LinObserver.manageFileInParam -> getSourceReference); a cache that lays the attempts of one task out side by side keeps the two-level <2hex>/<30hex> hierarchy and appends the attempt to the leaf. Without this the parser returns null for those inputs and lineage records them as bare DataPaths, losing the edge to the producing task and paying a content checksum in its place. The parser is already lenient by design -- a non-2-char bucket and an unparseable hash both return null -- so this is one more accepted leaf shape, and hex never contains a dash, so it stays unambiguous. Assisted-by: Claude Opus 5 (1M context) Signed-off-by: jorgee <jorge.ejarque@seqera.io>
jorgee
force-pushed
the
globalcache-core-spi
branch
from
September 18, 2026 11:25
98dd386 to
671d20d
Compare
It required NXF_GCS_TEST_BUCKET and GOOGLE_PROJECT_ID, neither of which is set anywhere, so it never ran. AwsS3BaseSpec and AzBaseSpec create and delete their own bucket from credentials alone; this does the same, and takes the project from the credentials too -- GoogleOpts.getProjectIdFromCreds reads project_id out of the service-account JSON that GOOGLE_APPLICATION_CREDENTIALS points at. So the gate is now credentials alone, matching BatchLoggingTest. A user ADC from `gcloud auth application-default login` carries no project_id, and getProjectIdFromCreds throws rather than returning null for one that does not, so that case is caught and skipped: a developer without a service-account key is not stopped by a test that can still run where such a key exists. Assisted-by: Claude Opus 5 (1M context) Signed-off-by: jorgee <jorge.ejarque@seqera.io>
jorgee
marked this pull request as ready for review
September 18, 2026 11:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds four extension points and their cloud implementations. Every default preserves current behaviour exactly — with no extension registered, the code takes the path it takes today.
nextflow.processor— task hashing and cache resolutionTaskHasherFactorycreates theTaskHasherfor a task, or abstains. With none registered, or all abstaining,TaskProcessordoesnew TaskHasher(task)as before; core'sTaskHasheris untouched.TaskCacheStrategydecides how a task is resolved against the cache, through theTaskResolverprimitivesTaskProcessorimplements.DefaultTaskCacheStrategyis today'scheckCachedOrLaunchTaskloop moved verbatim — thetriesfold, the lock,mkdirs,submitTask— andcheckCachedOrLaunchTaskbecomes the dispatch that picks it.nf-commons— object-storage capabilitiesObjectStoreReaderadds a flat, no-delimiter listing returning each object's size and mtime, and a ranged read of a single byte window. The listing costsO(N/1000)calls regardless of nesting depth, where the NIO walk costs one delimiter-based LIST per subdirectory. Both methods default tonull, so a scheme without a provider falls back to the hierarchical walk and the full read.AtomicLockProvideradds create-if-absent of a marker object, answered by the store rather than by a read-then-write. Object storage has no directory-create to race on, so this is the portable way for two processes on different machines to agree on which owns a prefix.Implemented for S3 (
ifNoneMatch("*")), Azure Blob (setIfNoneMatch('*'), matched on the error code rather than the status, with the six responses that mean "taken" driven from a table) and GCS (doesNotExist()). None is a read-then-write.Supporting changes
CacheFactorymay resolve its own work directory and write it back to the session, soSession.initcreates the cache before readingworkDir— everything below it must see the effective value, orworkflow.workDirnames a directory no task uses. A throw after that point now closes the cache instead of leaking it.CacheStore.updateEntry(a default method) lets a store composed of several members route an update back to the member that served the read.CacheDB.dispatchWriteis an internal refactor so a failed async cache write is logged with its cause rather than swallowed.FileHelper.getTaskHashFromPathtolerates an optional-Nsuffix on the work-directory leaf.nf-lineageresolves every task input to its producing task by parsing that path (LinObserver.manageFileInParam→getSourceReference). A cache that lays the attempts of one task out side by side keeps the two-level<2hex>/<30hex>hierarchy and appends the attempt number to the leaf; without this tolerance the parser returnsnullfor those inputs, and lineage records them as bareDataPathentries — losing the edge to the producing task and paying a content checksum in its place. The parser is already lenient by design (a non-2-char bucket and an unparseable hash both returnnull), so this is one more accepted leaf shape, and hex never contains a dash, so it stays unambiguous.