Record Slurm submit failures in the task stderr file - #1252
Open
stark256-spec wants to merge 1 commit into
Open
Conversation
When sbatch fails at submission time, Slurm never starts the job and so never writes the task's .err file, leaving no record in the workflow directory of what caused the failure (issue lanl#685). Capture the sbatch stderr on a submit failure and append it to the same path the task's stderr would have used, where a user would naturally look. The write is best-effort so it can't mask the original error. Adds worker_utils.write_submit_failure() and unit tests covering the helper (create + append) and the submit_job failure path. Closes lanl#685
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #685.
Problem
When a task ends with a submit failure in Slurm (
sbatchreturns non-zero), Slurm never starts the job and therefore never writes the task's.errfile in the workflow directory. The submit failure reason is raised as aWorkerError, but there is no persistent record in the workflow directory of what caused the failure, so a user inspecting the run afterwards has nothing to look at.Fix
On a submit failure, capture the
sbatchstderr and append it to the same path the task's stderr would have used (as resolved byresolve_stdout_stderr) — the natural place a user looks for a task's error output. TheWorkerErroris still raised as before.worker_utils.write_submit_failure(stderr_path, message): appends"Slurm job submission failed:\n<message>"to the task's stderr path. It is best-effort — anOSErrorwhile writing is logged rather than raised, so it can never mask the original submit error.BaseSlurmWorker.submit_jobcalls it before raising whensbatchreturns a non-zero exit code.This addresses @pagrubel's note on the issue about giving the failure somewhere to live; writing to the task's
.errpath means it lands where the user already expects task error output rather than in separate metadata.Tests
Added to
beeflow/tests/test_slurm_worker.py(all runnable without a live Slurm/slurmrestd):test_write_submit_failure_creates_record— the reason is written to the stderr path.test_write_submit_failure_appends— writing appends rather than truncating existing output.test_submit_job_records_failure— a failedsbatch(mocked) makessubmit_jobraiseWorkerErrorand leaves the reason in the task's resolved.errfile.pylint --rcfile=setup.cfgon the changed files: 10.00/10 (source), and the new test code adds no new warnings.