Throttle slurmdb usage on job-exit email creation...

[gavila1-brown]

Hi Guys,

We seem to be running into an issue that occurs when jobstats sends email after a job exits; especially when large numbers of jobs exit simultaneously. There seems to be a substantial load on the slurm database; so much so that commands like sacctmgr will hang while jobstats is loading data after exit-tbh, it doesn't seem like the database insertion is the issue, it's the query done when sending the job performance email, if I'm seeing this correctly. Is there a way to throttle jobstats DB access; perhaps make sending the job emails asynchronously?

Thanks!!

-Jeff

[plazonic]

Hello,

interesting and a bit puzzling. I assume you are using slurmctld epilog script to store job info in AdminComment - if so that should trigger two interactions with the slurm database. The script will run jobstats command to get the hash and jobstats will run sacct to get basic job info for that specific job (like start,end,nodes,state, ....). Since it is querying that specific job and therefore there is no searching within the database this should be fairly efficient and quick.

The second call is to set the AdminComment field. Again, should be fairly efficient.

Are you seeing an increase in db load? E.g. can you see lots of sacct processes hanging when this happens? If so how many?

Otherwise I'd expect more trouble from prometheus. If these are large and/or long running jobs getting stats out of prometheus can take a bit of time.

[gavila1-brown]

So it looks like 30-40 ors so sacct commands piling up when it has the issue. Thanks,

…

-Jeff

On Apr 29, 2025, at 10:32 AM, Josko Plazonic ***@***.***> wrote: plazonic left a comment (PrincetonUniversity/jobstats#29) <#29 (comment)> Hello, interesting and a bit puzzling. I assume you are using slurmctld epilog script to store job info in AdminComment - if so that should trigger two interactions with the slurm database. The script will run jobstats command to get the hash and jobstats will run sacct to get basic job info for that specific job (like start,end,nodes,state, ....). Since it is querying that specific job and therefore there is no searching within the database this should be fairly efficient and quick. The second call is to set the AdminComment field. Again, should be fairly efficient. Are you seeing an increase in db load? E.g. can you see lots of sacct processes hanging when this happens? If so how many? Otherwise I'd expect more trouble from prometheus. If these are large and/or long running jobs getting stats out of prometheus can take a bit of time. — Reply to this email directly, view it on GitHub <#29 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMOVW5KEX7QZH6HJM2K26HT236ERHAVCNFSM6AAAAAB4DE5O72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMZZGE3DEMRUHA>. You are receiving this because you authored the thread.

[plazonic]

Just to comment that I'll try to find time to work on this in the coming months - with some of the recent changes in slurm even sites that don't have a load problem with prometheus would probably find it useful to address this somehow.

The idea would be to use some kind of queuing mechanism that would post-process slurmctld epilog requests. The slurmctld epilog would submit a jobid to process and exit immediately. MailProg, if configured to send a jobstats summary, would have to do the same thing (otherwise it would run jobstats anyway and load prometheus).