Add parallel intitialization for TLSPH arrays to ensure NUMA awareness [Alternative] - #1294
Add parallel intitialization for TLSPH arrays to ensure NUMA awareness [Alternative]#1294efaulhaber wants to merge 5 commits into
Conversation
efaulhaber
left a comment
There was a problem hiding this comment.
Codex Review
The PR replaces sequential copies of TLSPH runtime arrays with parallel initialization through ThreadedBroadcastArray and PolyesterBackend to improve NUMA first-touch placement. No concrete correctness or regression issues were found in this pass.
This is an AI-generated code review. Please verify the findings and summary before acting on them.
Review generated by codex-pr-review
efaulhaber
left a comment
There was a problem hiding this comment.
Codex Review
The PR intends to improve NUMA performance by parallelizing first-touch initialization of TLSPH runtime arrays.
This is an AI-generated code review. Please verify the findings and summary before acting on them.
Review generated by codex-pr-review
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1294 +/- ##
==========================================
- Coverage 90.39% 90.38% -0.01%
==========================================
Files 138 138
Lines 11322 11341 +19
==========================================
+ Hits 10234 10251 +17
- Misses 1088 1090 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This is a simpler alternative to #1256. I dislike that we use
PolyesterBackendin this PR even if the user specifiedSerialBackendor a GPU backend (and doesn't expect us to use multiple threads), but the alternative is to initialize the arrays ininitialize!, and that leads to a multitude of problems becauseinitial_coordinatesis accessed in several places beforeinitialize!is called (see #1256). So I think this PR is still the cleaner solution.Benchmark on 2 x AMD EPYC 9965 (192 cores each) with 8 NUMA domains (48 cores each):

We can nicely see how this massive CPU is slow for small problems (when using all 384 cores), reaches maximum performance around 250k particles, then drops again when the problem size exceeds the cache, and then stagnates limited by the memory bandwidth.
The effect is much more pronounced for the deformation gradient:

With serial initialization, all data lives on one NUMA domain, decreasing memory locality and bandwidth. With parallel initialization, data lives on the NUMA domain belonging to the core that handles the corresponding particle.