More elegantly handle SIGINT

[c-prather]

Currently, Parthenon has custom handlers for SIGINT, SIGTERM, and SIGALRM, which all set a flag, wait until the end of the next step, output any final files, and then terminate the program (a 'clean exit', as opposed to immediate termination).

This PR adds a fallback to exit immediately if more than three SIGINTs or SIGTERMs are received before the next file output -- i.e., if the program is hung, and the user is repeatedly pressing ^C. It will also add a trigger file stop_now which triggers a clean exit upon discovering its existence, just like the current output_now file.

PR Summary

This makes the "signalflag" array in SignalHandler into a counter of signals received since the last reset. It then adds intelligence to SetSignalFlag to throw an error if some number (currently 3) SIGINTs have been received since the flag was last cleared.

The number/behavior is not currently runtime-configurable -- I think that so long as we only die on 3+ SIGINT we're just never triggered by an MPI runtime (acting alone).

However, there are some runtimes in which it's impossible to send one SIGINT without ungracefully killing the program. Thus I'm also planning to monitor for a stop_now file.

(For reference: some frontends (srun) eat the first SIGINT and do nothing, whereas others (OpenMPI) effectively do nothing on 2 or more SIGINTs as they exit immediately themselves upon receiving the second. In fact, OpenMPI will never actually send a SIGINT specifically -- when it receives one, it upgrades it to a SIGTERM before sending it on, and after a timer sends SIGKILL.)

Thus we will have a general trigger:

• ^C to elegantly stop and write last outputs, if possible
• Repeated ^C to kill the program without cleanup
• stop_now to request an elegant stop, but without any power to actually kill the program

PR Checklist

• Code passes cpplint
• New features are documented.
• Adds a test for any bugs fixed. Adds tests for new features.
• Code is formatted
• Changes are summarized in CHANGELOG.md
• Change is breaking (API, behavior, ...)
  • Change is additionally added to CHANGELOG.md in the breaking section
  • PR is marked as breaking
  • Short summary API changes at the top of the PR (plus optionally with an automated update/fix script)
• CI has been triggered on Darwin for performance regression tests.
• Docs build
• Any contribution that was created or modified with the assistance of generative AI is disclosed here and in code following the guidelines
• (@lanl.gov employees) Update copyright on changed files

[c-prather]

@Yurlungur @nbowden045 here's the updated version of the SIGINT handling PR. Basically: it's sometimes impossible to make your MPI runtime forward one SIGINT. But, I think we can still behave consistently under most circumstances: repeated SIGINT (and overzealous MPI implementations) just cause the program to die, but add a touch/trigger file stop_now which will request an elegant stop no matter how long it takes.

[Yurlungur]

We still try to die gracefully after the first signal, right?

[c-prather]

Yes! Until the counter hits 3 Parthenon has still just set the flag to end after the next step.

To be clear, I still need to implement the file option, and test it. Not sure if our regression infra supports "send signal/create file after N seconds" but I could add a test if so.

[pgrete]

So far so good. Looking forward to the exit_now trigger (and maybe die_now to kill the sim without outputs).

[pgrete]

Do you also plan to use the counter here or why is the "+"=1 important?

[pgrete]

Same here.

[pgrete]

Suggested change

	- [[PR 782]](https://github.com/parthenon-hpc-lab/parthenon/pull/782) Die on repeatedly receiving SIGINT
	- [[PR 1446]](https://github.com/parthenon-hpc-lab/parthenon/pull/1446) Die on repeatedly receiving SIGINT