Fix output index file write to object storage - #7635
Merged
Merged
Conversation
Write CSV index files in a single operation instead of appending line by line. The Google Cloud Storage NIO provider does not support APPEND, so writing an index file to a `gs://` output directory failed with `UnsupportedOperationException: APPEND`. The JSON and YAML branches were unaffected because they already write the file in one go. Closes #7633 Signed-off-by: Ben Sherman <bentshermann@gmail.com>
✅ Deploy Preview for nextflow-docs canceled.
|
jorgee
approved these changes
Sep 21, 2026
25 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7633
Problem
Writing a CSV index file from an
outputblock fails when the output directory is ags://bucket:CsvWriter.apply()built the file with Groovy'spath << textoperator, which opens the file withStandardOpenOption.APPEND. GCS objects are immutable once written, so its NIO provider rejects that option outright.The issue was reported against
google-batch, but the executor is irrelevant. The index file is written by the head job, so any run with ags://output directory hits this, local executor included. Settingindex.pathto a local path does not help either, sincePublishOp.onComplete()always resolves it againstsession.outputDir.Only the CSV branch was affected. JSON and YAML index files already assign
path.text = ..., a single write.Change
Build the CSV in a
StringBuilderand write it once. Index files are a list of published outputs, small enough that buffering costs nothing.Two lines dropped as redundant: the leading
path.delete()(assigningpath.texttruncates) and theif( records.isEmpty() ) path << ''case (assigning an empty string already creates the file).Testing
CsvWriterTestpasses unchanged, including the empty-file case and the test that writes the same path twice.Verified against a live GCS bucket. Before the change the run aborted with the stack trace above; after it, the object is written and reads back correctly:
Note that
CsvWriterTestuses a Jimfs in-memory filesystem, which permits append, so the existing tests could never have caught this. A regression test would need a filesystem that rejectsAPPEND. I have left that out of this PR, happy to add it if reviewers want it.Unrelated observation
The failed run printed
[SUCCESS]and exited 0 despiteonCompletethrowing, so a broken index write is silent in CI. Separate issue, not addressed here.