Add pipeline stop; stop hiding jobs in job list - #61
Merged
Merged
Conversation
Two of the four defects in the pipeline-run-lifecycle report. Both are about not being able to see or stop what a run left behind. `abc pipeline stop <run>` stops a run and every worker job it submitted. Stopping the head does not stop a run: the head has already submitted its workers to Nomad, which keeps running them and keeps starting queued ones as capacity frees. Draining that by hand took repeated rounds of grepping `abc infra compute show` and stopping seven at a time while the next seven started. The head and every worker share a `<user>-<runid>` prefix, so the set is addressable in one operation. The argument accepts the head id, any worker id, or the bare tag. The head is stopped first so it cannot submit more work during the drain. Script jobs are rejected rather than prefix-matched. Definitions are kept by default — the server disables GC deliberately so run records stay queryable — with `--purge` to remove them, plus `--dry-run`. Verified on seedling-prod against a real orphan: run abhinav-1788623348 had a dead head and seven workers still pending. `pipeline stop` found all seven and cleared them; a second call reports nothing running. `job list` was hiding most jobs. It fetched every job, then truncated to --limit (default 20) in Nomad's own ID-ascending order. Pipeline jobs are named `abhinav-<runid>-…`, which sorts after `abc-…` and `abhin-script-job-…`, so they were never visible at the default limit however recent — `abc job show` would find a running job that `abc job list | grep` reported nothing for. On this cluster the default view showed 20 of 293, and `abc job list | grep -c SELECT_` returned 0 while 97 such jobs existed. Now filtered, then sorted newest-first, then limited, and it says when the view is partial. The report read this as `list` and `show` disagreeing; it was silent truncation.
nf-nomad stamps the correlation onto every worker job it submits:
nf_head_job_id abhinav-1788623348-nf-head-nf-core-demo
nf_head_alloc_id 29d5183f-93f7-5a33-cee8-38106d017d24
nf_session_name abhinav-1788623348
abc_run_name abhinav-1788623348-nf-head-nf-core-demo
Present on 25/25 sampled workers on seedling-prod. That is authoritative;
the name prefix is a cheap way to narrow candidates, not a membership test.
Two things it fixes:
- a neighbouring run whose id merely shares a prefix (1788623348 vs
17886233480) is now rejected instead of swept up;
- the head is identified by nf_head_job_id rather than by the "-nf-head-"
substring, which was guesswork about a naming convention.
The job LIST stub carries no Meta, so this costs one GetJob per candidate,
bounded by the size of a single run. A job carrying no meta at all falls back
to the name match rather than being dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses §3 and §4 of
brainstorms/abc-cluster-cli/2026-09-05-pipeline-run-lifecycle-and-config-precedence.md.§3 —
abc pipeline stop <run>Stopping the head does not stop a run. The head has already submitted its
worker jobs to Nomad, which keeps running them and keeps starting queued ones
as capacity frees. The report describes draining that by hand in more than a
dozen rounds, seven at a time, with the next seven starting immediately.
The new command stops the head first — so it cannot submit more work during
the drain — then the workers. It takes the head id, any worker id, or the bare
run tag.
Definitions are kept by default, since the server disables GC deliberately so
run records stay queryable;
--purgeremoves them,--dry-runshows the set.Membership comes from job meta, not the name. nf-nomad stamps the
correlation onto every worker:
Present on 25/25 sampled workers. The name prefix only narrows candidates
cheaply (the list stub carries no
Meta, so confirming costs oneGetJobpercandidate, bounded by one run's size). This rejects a neighbouring run sharing
an id prefix, and identifies the head by what it declares rather than by the
-nf-head-substring. Script jobs are rejected outright rather thanprefix-matched.
Verified on seedling-prod against a real orphan. Run
abhinav-1788623348had a dead head and seven workers still pending — exactly the report's
scenario.
pipeline stopfound all seven and cleared them; a second callreports nothing running. They stayed stopped without
--purge.§4 —
job listwas hiding most jobsIt fetched every job, then truncated to
--limit(default 20) in Nomad's ownID-ascending order. Pipeline jobs are named
abhinav-<runid>-…, which sortsafter
abc-…andabhin-script-job-…, so they were never visible at thedefault limit however recent.
That is what made
showandlistlook inconsistent. On this cluster:job list | grep -c SELECT_Now filtered, then sorted newest-first, then limited, with a line stating when
the view is partial.
Not in this PR
§1 (
queueSizein the pipeline's config silently overridden by the generatedhead config) and §2 (a blocked placement reported as a task failure and
retried) are untouched.
Worth noting on §2: the quoted symptom is
WARN: [NOMAD] Cannot determine exit status for task, which is the nf-nomaddefect fixed in nextflow-io/nf-nomad#148, and placement classification was
fixed in #145 (released as 0.5.0-edge4). Some of §2 may already be addressed
plugin-side once the cluster picks up a release carrying both.
Full suite green.