Skip to content

Fix output index file write to object storage - #7635

Merged
bentsherman merged 2 commits into
masterfrom
fix-csv-index-append
Sep 21, 2026
Merged

bentsherman merged 2 commits into
masterfrom
fix-csv-index-append

Conversation

@bentsherman

Copy link
Copy Markdown
Member

Closes #7633

Problem

Writing a CSV index file from an output block fails when the output directory is a gs:// bucket:

java.lang.UnsupportedOperationException: APPEND
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newWriteChannel(CloudStorageFileSystemProvider.java:461)
	at org.apache.groovy.nio.extensions.NioExtensions.leftShift(NioExtensions.java:500)
	at nextflow.util.CsvWriter.apply(CsvWriter.groovy:43)
	at nextflow.extension.PublishOp.onComplete(PublishOp.groovy:236)

CsvWriter.apply() built the file with Groovy's path << text operator, which opens the file with StandardOpenOption.APPEND. GCS objects are immutable once written, so its NIO provider rejects that option outright.

The issue was reported against google-batch, but the executor is irrelevant. The index file is written by the head job, so any run with a gs:// output directory hits this, local executor included. Setting index.path to a local path does not help either, since PublishOp.onComplete() always resolves it against session.outputDir.

Only the CSV branch was affected. JSON and YAML index files already assign path.text = ..., a single write.

Change

Build the CSV in a StringBuilder and write it once. Index files are a list of published outputs, small enough that buffering costs nothing.

Two lines dropped as redundant: the leading path.delete() (assigning path.text truncates) and the if( records.isEmpty() ) path << '' case (assigning an empty string already creates the file).

Testing

CsvWriterTest passes unchanged, including the empty-file case and the test that writes the same path twice.

Verified against a live GCS bucket. Before the change the run aborted with the stack trace above; after it, the object is written and reads back correctly:

"id","value"
"sample_1","1"
"sample_2","2"
"sample_3","3"

Note that CsvWriterTest uses a Jimfs in-memory filesystem, which permits append, so the existing tests could never have caught this. A regression test would need a filesystem that rejects APPEND. I have left that out of this PR, happy to add it if reviewers want it.

Unrelated observation

The failed run printed [SUCCESS] and exited 0 despite onComplete throwing, so a broken index write is silent in CI. Separate issue, not addressed here.

Write CSV index files in a single operation instead of appending line by
line. The Google Cloud Storage NIO provider does not support APPEND, so
writing an index file to a `gs://` output directory failed with
`UnsupportedOperationException: APPEND`. The JSON and YAML branches were
unaffected because they already write the file in one go.

Closes #7633

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@netlify

netlify Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for nextflow-docs canceled.

Name Link
🔨 Latest commit 29b59e0
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs/deploys/6ab1315eb5f0250008d2d6af

@bentsherman
bentsherman merged commit 0c671cb into master Sep 21, 2026
24 checks passed
@bentsherman
bentsherman deleted the fix-csv-index-append branch September 21, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

can't write output index files using google-batch executor & gs storage bucket for output dir

2 participants