WIP: Performance Improvements (for Burgers)

[fglines-nv]

PR Summary

Set of optimizations targeting the burgers benchmark but may improve/impact other downstream codes. Optimizations include an MPI_Barrier after the MPI_ISends to force MPI libraries to start sends (this might be implementation dependent, it's applicable to OpenMPI/HPCX during testing); an object to specify team sizes for par_outer_for and potentially launch bounds in the future (for register pressured kernels); OMP parallelizations for some of the AMR machinery; some changes to burgers driver/package; and some checks disabled for burgers (but might be necessary for sparse variables).

Very WIP, I'd like to replace the #pragma omp ... with something more performance portable and durable. I was compiling with -fopenmp instead of Kokkos_ENABLE_OMP, which does many other things that apparently slow down the benchmark. Not ideal, suggestions welcome on how to insert OMP parallelization on host-side loops for the GPU-resident path. Also needs testing on everything not-burgers.

PR Checklist

• Code passes cpplint
• New features are documented.
• Adds a test for any bugs fixed. Adds tests for new features.
• Code is formatted
• Changes are summarized in CHANGELOG.md
• Change is breaking (API, behavior, ...)
  • Change is additionally added to CHANGELOG.md in the breaking section
  • PR is marked as breaking
  • Short summary API changes at the top of the PR (plus optionally with an automated update/fix script)
• CI has been triggered on Darwin for performance regression tests.
• Docs build
• Any contribution that was created or modified with the assistance of generative AI is disclosed here and in code following the guidelines
• (@lanl.gov employees) Update copyright on changed files

[fglines-nv]

I used AI to refactor a human-written optimized code another engineer had written to reduce the amount of repeated code. Originally there were 4x versions of this loop, AI reduced it down to 2 versions (but only 100 fewer lines).

The new functions PassesFluxCorrectionFilter and GetBoundaryAtIndex were granulized out from the original and the ForEachBoundary* functions refactored to use them. The original strategy is more or less unchanged except AI opted to create the boundary index in a single pass with vector::push_back instead of two passes, one to count and one to create.