Skip to content

Record Slurm submit failures in the task stderr file - #1252

Open
stark256-spec wants to merge 1 commit into
lanl:developfrom
stark256-spec:fix/slurm-submit-failure-stderr
Open

Record Slurm submit failures in the task stderr file#1252
stark256-spec wants to merge 1 commit into
lanl:developfrom
stark256-spec:fix/slurm-submit-failure-stderr

Conversation

@stark256-spec

Copy link
Copy Markdown

Closes #685.

Problem

When a task ends with a submit failure in Slurm (sbatch returns non-zero), Slurm never starts the job and therefore never writes the task's .err file in the workflow directory. The submit failure reason is raised as a WorkerError, but there is no persistent record in the workflow directory of what caused the failure, so a user inspecting the run afterwards has nothing to look at.

Fix

On a submit failure, capture the sbatch stderr and append it to the same path the task's stderr would have used (as resolved by resolve_stdout_stderr) — the natural place a user looks for a task's error output. The WorkerError is still raised as before.

  • New worker_utils.write_submit_failure(stderr_path, message): appends "Slurm job submission failed:\n<message>" to the task's stderr path. It is best-effort — an OSError while writing is logged rather than raised, so it can never mask the original submit error.
  • BaseSlurmWorker.submit_job calls it before raising when sbatch returns a non-zero exit code.

This addresses @pagrubel's note on the issue about giving the failure somewhere to live; writing to the task's .err path means it lands where the user already expects task error output rather than in separate metadata.

Tests

Added to beeflow/tests/test_slurm_worker.py (all runnable without a live Slurm/slurmrestd):

  • test_write_submit_failure_creates_record — the reason is written to the stderr path.
  • test_write_submit_failure_appends — writing appends rather than truncating existing output.
  • test_submit_job_records_failure — a failed sbatch (mocked) makes submit_job raise WorkerError and leaves the reason in the task's resolved .err file.
$ pytest beeflow/tests/test_slurm_worker.py -k "write_submit or records_failure"
3 passed

pylint --rcfile=setup.cfg on the changed files: 10.00/10 (source), and the new test code adds no new warnings.

When sbatch fails at submission time, Slurm never starts the job and so
never writes the task's .err file, leaving no record in the workflow
directory of what caused the failure (issue lanl#685).

Capture the sbatch stderr on a submit failure and append it to the same
path the task's stderr would have used, where a user would naturally
look. The write is best-effort so it can't mask the original error.

Adds worker_utils.write_submit_failure() and unit tests covering the
helper (create + append) and the submit_job failure path.

Closes lanl#685
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slurm Submit Fail Doesn't Provide Error Output

1 participant