Questions about ingest_jobstats

[bsper2]

I noticed that the current ingest_jobstats script doesn't play nice when storing jobstats in an external DB, because it 1) only supports calling sacctmgr to set the AdminComment and 2) the list of jobs it gets from jobs_with_no_data.py only looks at jobs in the actual Slurm DB that have a null for AdminComment.

Did I miss something or is it accurate to say that clusters that are using an external DB should skip setting up the ingest_jobstats pieces?

If so I can open a Pull request and add a note about that to the docs if you'd like.

Also, I was curious how important the ingest_jobstats was. I can see it being really handy if you rolled out the Prometheus exporters and then it was a while before you got around to setting up slurmctldepilog.sh so you could get all those jobs that completed in-between processed. The docs mention it can be useful for when the slurmctldepilog fails, but it didn't seem like that would be happening too often. Is the ingest_jobstats just a nice-to-have or would I be really missing things in certain scenarios if I wasn't using it?

[Anish-Chanda]

Hey @bsper2 Thanks for creating this issue. That is a good catch, the ingest_jobstats script does not support an external DB right now. So yeah, with an external DB, you only get coverage for jobs when slurmctldepilog runs at completion.

But this is a rather small fix. We can make jobs_with_no_data check the external table instead of Slurm, depending on the config, and have ingest use store_jobstats.py when needed. After that, it should work transparently whichever mode you're in.

I will get a fix going for this soon.

To answer your other questions, for normal operation, it's a nice-to-have. Slurmctldepilog is pretty reliable. But it matters in two cases: backfilling old jobs during initial setup, and if the database goes down, epilog can't write to it. Without ingest in external DB mode, there's no way to catch up.

[bsper2]

Got it, sounds good thanks for the info!