Focus call-graph macrobenchmarks on analyses - #2053
Merged
Merged
Conversation
Add JMH macro-benchmarks for the call-graph workload that measure only pointer-analysis construction, not the test-harness verification the existing end-to-end benchmark includes. The dominant call-graph benchmark (`testHelloAllEntrypoints`) times a whole unit test; a JFR profile showed ~30% of its wall time is spent verifying the results (graph-integrity checks, call-graph squashing and subset assertions, stat printing) rather than in the analysis itself. That verification pollutes the timed signal with JIT'ed, IO-bound, and assertion work, inflates the AB-experiment cycle, and obscures what real WALA users pay for when building call graphs. Split the five-phase pipeline in `CallGraphTest.doCallGraphs` into a build step (all five call graphs, sharing one analysis cache) and the existing verification step. The verification remains unchanged and always runs in the test path, in the same phase order, so the graph-structure and determinism assertions are exactly as before. Add two kinds of benchmarks on top: * per-phase single-shot benchmarks (RTA, 0-CFA, 0-1-CFA, 0-Container-CFA, 0-1-Container-CFA) for cost attribution, with the scope, class hierarchy, options, and cache built once outside the timed region; * an ensemble benchmark that runs all five phases in sequence with no verification, preserving the real pipeline's cross-phase cache and JIT behavior. Every benchmark returns the built call graphs so the JIT cannot elide the work. No production code is touched. Impact using my personal development machine and Zulu Java 17: the ensemble benchmark runs in ~5.8 s versus ~8.1 s for the end-to-end probe, removing the ~30% verification noise and landing almost exactly on the profile's predicted ~70% analysis portion. The per-phase methods attribute cost across algorithms (0-1-Container-CFA dominates at ~1.9 s), giving AB experiments a cheaper, lower-variance, call-graph-specific number to track optimizations.
msridhar
approved these changes
Sep 10, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2053 +/- ##
=========================================
Coverage 50.80% 50.81%
Complexity 12760 12760
=========================================
Files 1368 1368
Lines 83500 83500
Branches 14414 14414
=========================================
+ Hits 42426 42427 +1
Misses 36388 36388
+ Partials 4686 4685 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add JMH macro-benchmarks for the call-graph workload that measure only pointer-analysis construction, not the test-harness verification the existing end-to-end benchmark includes.
The dominant call-graph benchmark (
testHelloAllEntrypoints) times a whole unit test; a JFR profile showed ~30% of its wall time is spent verifying the results (graph-integrity checks, call-graph squashing and subset assertions, stat printing) rather than in the analysis itself. That verification pollutes the timed signal with JIT'ed, IO-bound, and assertion work, inflates the AB-experiment cycle, and obscures what real WALA users pay for when building call graphs.Split the five-phase pipeline in
CallGraphTest.doCallGraphsinto a build step (all five call graphs, sharing one analysis cache) and the existing verification step. The verification remains unchanged and always runs in the test path, in the same phase order, so the graph-structure and determinism assertions are exactly as before.Add two kinds of benchmarks on top:
per-phase single-shot benchmarks (RTA, 0-CFA, 0-1-CFA, 0-Container-CFA, 0-1-Container-CFA) for cost attribution, with the scope, class hierarchy, options, and cache built once outside the timed region;
an ensemble benchmark that runs all five phases in sequence with no verification, preserving the real pipeline's cross-phase cache and JIT behavior.
Every benchmark returns the built call graphs so the JIT cannot elide the work. No production code is touched.
Impact using my personal development machine and Zulu Java 17: the ensemble benchmark runs in ~5.8 s versus ~8.1 s for the end-to-end probe, removing the ~30% verification noise and landing almost exactly on the profile's predicted ~70% analysis portion. The per-phase methods attribute cost across algorithms (0-1-Container-CFA dominates at ~1.9 s), giving AB experiments a cheaper, lower-variance, call-graph-specific number to track optimizations.