Skip to content

Add parallel intitialization for TLSPH arrays to ensure NUMA awareness [Alternative] - #1294

Open
efaulhaber wants to merge 5 commits into
trixi-framework:mainfrom
efaulhaber:tlsph-numa-awareness2
Open

Add parallel intitialization for TLSPH arrays to ensure NUMA awareness [Alternative]#1294
efaulhaber wants to merge 5 commits into
trixi-framework:mainfrom
efaulhaber:tlsph-numa-awareness2

Conversation

@efaulhaber

Copy link
Copy Markdown
Member

This is a simpler alternative to #1256. I dislike that we use PolyesterBackend in this PR even if the user specified SerialBackend or a GPU backend (and doesn't expect us to use multiple threads), but the alternative is to initialize the arrays in initialize!, and that leads to a multitude of problems because initial_coordinates is accessed in several places before initialize! is called (see #1256). So I think this PR is still the cleaner solution.

Benchmark on 2 x AMD EPYC 9965 (192 cores each) with 8 NUMA domains (48 cores each):
grafik
We can nicely see how this massive CPU is slow for small problems (when using all 384 cores), reaches maximum performance around 250k particles, then drops again when the problem size exceeds the cache, and then stagnates limited by the memory bandwidth.

The effect is much more pronounced for the deformation gradient:
grafik

With serial initialization, all data lives on one NUMA domain, decreasing memory locality and bandwidth. With parallel initialization, data lives on the NUMA domain belonging to the core that handles the corresponding particle.

@efaulhaber efaulhaber self-assigned this Aug 17, 2026

@efaulhaber efaulhaber left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex Review

The PR replaces sequential copies of TLSPH runtime arrays with parallel initialization through ThreadedBroadcastArray and PolyesterBackend to improve NUMA first-touch placement. No concrete correctness or regression issues were found in this pass.

This is an AI-generated code review. Please verify the findings and summary before acting on them.
Review generated by codex-pr-review

Comment thread src/schemes/structure/total_lagrangian_sph/system.jl
Comment thread src/schemes/structure/total_lagrangian_sph/system.jl Outdated
Comment thread src/schemes/structure/total_lagrangian_sph/system.jl

@efaulhaber efaulhaber left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex Review

The PR intends to improve NUMA performance by parallelizing first-touch initialization of TLSPH runtime arrays.

This is an AI-generated code review. Please verify the findings and summary before acting on them.
Review generated by codex-pr-review

Comment thread src/util.jl
Comment thread src/util.jl
Comment thread src/util.jl
@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.30435% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.38%. Comparing base (a720b1d) to head (104fe9a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/util.jl 85.71% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1294      +/-   ##
==========================================
- Coverage   90.39%   90.38%   -0.01%     
==========================================
  Files         138      138              
  Lines       11322    11341      +19     
==========================================
+ Hits        10234    10251      +17     
- Misses       1088     1090       +2     
Flag Coverage Δ
total 90.39% <91.30%> (-0.01%) ⬇️
unit 77.50% <91.30%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@efaulhaber
efaulhaber requested a review from svchb August 18, 2026 09:22
@efaulhaber
efaulhaber marked this pull request as ready for review August 18, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant