Skip to content

An unread exit code is unknown, not success - #148

Merged
abhi18av merged 2 commits into
masterfrom
fix/exitcode-unknown-not-zero
Sep 4, 2026
Merged

abhi18av merged 2 commits into
masterfrom
fix/exitcode-unknown-not-zero

Conversation

@abhi18av

@abhi18av abhi18av commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Problem

When a task's .exitcode file cannot be read, defineExitCode() falls back to
scanning the Nomad task events and returning the first integer exitCode it
finds. Nomad leaves ExitCode at its zero value on every event that is not a
task exit, so an allocation that fails before the task body runs looks like
this — captured from an image-pull failure on Nomad 1.11.2:

event type ExitCode message
Received 0 Task received by client
Task Setup 0 Building Task Directory
Driver 0 Downloading image
Driver Failure 0 Failed to pull image
Not Restarting 0 Policy allows no restarts

ClientStatus=failed, Failed=true, and no .exitcode written anywhere. The
scan returns 0 from Received.

The caller reads a 0 as the worker reporting success and uses it to override
Nomad's own alloc-state failure. Nextflow is therefore told the task succeeded,
looks for outputs that were never produced, and raises:

Missing output file(s) `ri_summary.tsv` expected by process `CHECK_DB`

That message points at the process script, which was correct. The real failure —
an allocation that never started — had already been reported by Nomad and was
discarded.

This is not a race. Any allocation failing before the task body runs is read as
success, deterministically.

The comment above the caller claimed a missing exit file could not spuriously
suppress, "because defineExitCode returns Integer.MAX_VALUE when it couldn't
read any signal". That holds only if the events fallback does not exist.

Fix

Only accept a code from an event that represents the task terminating, which is
the sole case where Nomad populates ExitCode with a real value. Everything
else falls through to MAX_VALUE, so the failure propagates and
failureMessage surfaces the allocation's own events.

The genuine case the fallback served still works: a worker killed before it
could write .exitcode, where Nomad recorded the real code on Terminated
(the existing exitCode: 143 OOM coverage still passes unchanged).

Also tracks where the value came from, so the warning stops attributing an
events-derived code to local .exitcode. That mislabelling is why the original
report read as self-contradictory: the log said the file was unreadable and then
quoted a value from it.

Effect

Same pipeline, same cluster, one process whose container image does not exist:

Before

ERROR ~ Error executing process > 'NEVER_RUNS'
Caused by:
  Missing output file(s) `ri_summary.tsv` expected by process `NEVER_RUNS`
exit: 0

After

WARN  [NOMAD] Cannot determine exit status for task: `NEVER_RUNS`
ERROR ~ Error executing process > 'NEVER_RUNS'
Caused by:
  [NOMAD] Task failed in Nomad state dead. Task received by client |
  Building Task Directory | Downloading image |
  Failed to pull `abc-cluster/this-image-does-not-exist:none`: Error response
  from daemon: pull access denied … | Policy allows no restarts
  [NOMAD] Inspect job 'nf-d54beae7-…-NEVER_RUNS', allocation 'b6f10031-…',
  node 'client_dev'. Allocation API: http://127.0.0.1:4646/v1/allocation/…

Tests

Two cases, both using the real five-event sequence above:

  • defineExitCode returns MAX_VALUE rather than 0 when the allocation never ran
  • the reconciliation surfaces the Nomad failure instead of suppressing it, and the
    error names the pull failure

The first commit adds them as reproducers failing against current behaviour; the
second fixes it.

Full suite: 455 tests, 0 failures.

defineExitCode() falls back to scanning Nomad task events for the first
integer exitCode when the .exitcode file cannot be read. Nomad leaves
ExitCode at its zero value on every event that is not a task exit, so an
allocation that fails before the task body runs yields 0 from `Received`.

Captured from a live cluster (Nomad 1.11.2), image-pull failure:

    Received         ExitCode 0   Task received by client
    Task Setup       ExitCode 0   Building Task Directory
    Driver           ExitCode 0   Downloading image
    Driver Failure   ExitCode 0   Failed to pull image
    Not Restarting   ExitCode 0   Policy allows no restarts

ClientStatus=failed, Failed=true, and no .exitcode written.

The caller then reads that 0 as the worker reporting success and uses it to
override Nomad's own alloc-state failure, so Nextflow looks for outputs that
were never produced and reports MissingFileException against a process script
that was correct.

The existing coverage passes only because its fixture carries a single
Terminated event with the real code. This test uses the real sequence and
fails on current behaviour.

Not fixed here: this commit is the reproducer.
defineExitCode() fell back to scanning Nomad task events for the first
integer exitCode whenever the .exitcode file could not be read. Nomad
leaves ExitCode at its zero value on every event that is not a task exit,
so an allocation that failed before the task body ran returned 0 from
`Received`:

    Received         ExitCode 0   Task received by client
    Task Setup       ExitCode 0   Building Task Directory
    Driver           ExitCode 0   Downloading image
    Driver Failure   ExitCode 0   Failed to pull image
    Not Restarting   ExitCode 0   Policy allows no restarts

The caller reads a 0 as the worker reporting success and uses it to
override Nomad's own alloc-state failure. Nextflow was therefore told the
task succeeded, looked for outputs that were never produced, and raised
MissingFileException against a process script that was correct. The real
failure — an allocation that never started — had been reported by Nomad
and discarded.

The comment above the caller claimed a missing exit file could not
spuriously suppress, because defineExitCode returns Integer.MAX_VALUE when
it reads nothing. That held only if the events fallback did not exist.

Now only an event representing the task terminating is accepted, which is
the sole case where Nomad populates ExitCode with a real value. Everything
else falls through to MAX_VALUE, so the failure propagates and
failureMessage surfaces the allocation's own events. The genuine case the
fallback served — a worker killed before it could write .exitcode, where
Nomad recorded the real code on Terminated — still works.

Also tracks where the value came from, so the warning stops attributing an
events-derived code to `local .exitcode`. That is why the original report
read as self-contradictory: the log said the file was unreadable and then
quoted a value from it.
@abhi18av
abhi18av merged commit e9bd9cd into master Sep 4, 2026
3 checks passed
@abhi18av
abhi18av deleted the fix/exitcode-unknown-not-zero branch September 4, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant