fix(google-batch): recover the job when the createJob response is lost

[bentsherman]

Closes #6916

Problem

A run on google-batch aborts with an ALREADY_EXISTS error naming a job that Nextflow itself just generated:

Caused by:
  ALREADY_EXISTS: Resource "projects/.../jobs/nf-06071afa-1773255240303" already exists

Nothing external created that job — Nextflow collided with its own earlier create request.

BatchClient.apply() wraps every Batch API call in a failsafe retry policy that retries UnavailableException, DeadlineExceededException, IOException and TimeoutException. Every one of those is ambiguous: the request may have been applied server-side before the response was lost. createJob is not idempotent under such a retry, so the second attempt is rejected with ALREADY_EXISTS, which is not in the retry predicate and propagates out of submitJob() → GoogleBatchTaskHandler.submit() → TaskPollingMonitor, aborting the session.

The job it complains about keeps running and billing while the run is torn down. A reporter on the issue confirmed this with Cloud Logging: a 502:Bad Gateway, then ALREADY_EXISTS on the same job name, and gcloud batch describe showing that job eventually SUCCEEDED.

process.errorStrategy does not help, since a gax ApiException is not a ProcessException and so never reaches checkErrorStrategy().

This was flagged as a residual gap while triaging the sibling issue #6878; the DeadlineExceededException branch added in 944f48f adds another path into it but did not create it — UNAVAILABLE already reached it on 24.04.x.

Change

Handle AlreadyExistsException in submitJob() by fetching the job the previous attempt created. The job id is fixed per task handler (nf-<task hash>-<currentTimeMillis>, GoogleBatchTaskHandler:135), so ALREADY_EXISTS on that name can only mean an earlier attempt of the same submission succeeded.

The lookup sits inside the retry policy deliberately: if it comes back NOT_FOUND — ALREADY_EXISTS was reported but the job is not actually there — NotFoundException is already retryable, so failsafe re-runs the supplier and re-attempts the create. Wrapping apply() from the outside would retry only the lookup and then abort the task.

The recovery is scoped to submitJob; apply() stays generic, so ALREADY_EXISTS from getJob/listTasks/deleteJob still surfaces as before.

Tests

New BatchClientSubmitJobTest. BatchServiceClient.createJob and getJob are final and cannot be mocked, and BatchServiceGrpc is not on the classpath (google-cloud-batch builds its method descriptors by hand; only proto-google-cloud-batch-v1 is pulled in), so there is no generated service base to subclass either. The test instead injects a fake ManagedChannel that answers RPCs from a scripted queue and builds a real BatchServiceClient over it via FixedTransportChannelProvider — exercising the actual API client, the gax call plumbing and the failsafe policy, with no new dependency.

Six cases: plain submit; recovery after UNAVAILABLE; recovery after DEADLINE_EXCEEDED; NOT_FOUND on the lookup falling back to a fresh create; the bounded give-up path; and ALREADY_EXISTS still surfacing from a generic call.

Verified red/green — with the fix reverted, the four recovery cases fail with com.google.api.gax.rpc.AlreadyExistsException (the reported error) while the two control cases still pass. With it applied all six pass, and the full :plugins:nf-google:test suite is green.

Notes for review

• With a custom jobName closure a user can produce colliding names across concurrent runs, and this makes Nextflow adopt the pre-existing job rather than failing. The previous behaviour aborted the whole session in that case, so it was not a useful guard either — but if a louder signal is wanted, log.warn when the job's createTime predates this submit would cover it.
• Separate and not addressed here: createBatchService() builds BatchServiceSettings without any RetrySettings, so every call is pinned to gax's 60s totalTimeout and google.batch.requestTimeout (which several users set) is not a real config option.

[netlify]

✅ Deploy Preview for nextflow-docs canceled.

Name	Link
🔨 Latest commit	5c36528
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6aa49264f24f7a0008271ba1