Skip to content

Focus call-graph macrobenchmarks on analyses - #2053

Merged
liblit merged 1 commit into
wala:masterfrom
liblit:perf/callgraph-analysis-benchmarks
Sep 10, 2026
Merged

liblit merged 1 commit into
wala:masterfrom
liblit:perf/callgraph-analysis-benchmarks

Conversation

@liblit

@liblit liblit commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Add JMH macro-benchmarks for the call-graph workload that measure only pointer-analysis construction, not the test-harness verification the existing end-to-end benchmark includes.

The dominant call-graph benchmark (testHelloAllEntrypoints) times a whole unit test; a JFR profile showed ~30% of its wall time is spent verifying the results (graph-integrity checks, call-graph squashing and subset assertions, stat printing) rather than in the analysis itself. That verification pollutes the timed signal with JIT'ed, IO-bound, and assertion work, inflates the AB-experiment cycle, and obscures what real WALA users pay for when building call graphs.

Split the five-phase pipeline in CallGraphTest.doCallGraphs into a build step (all five call graphs, sharing one analysis cache) and the existing verification step. The verification remains unchanged and always runs in the test path, in the same phase order, so the graph-structure and determinism assertions are exactly as before.

Add two kinds of benchmarks on top:

  • per-phase single-shot benchmarks (RTA, 0-CFA, 0-1-CFA, 0-Container-CFA, 0-1-Container-CFA) for cost attribution, with the scope, class hierarchy, options, and cache built once outside the timed region;

  • an ensemble benchmark that runs all five phases in sequence with no verification, preserving the real pipeline's cross-phase cache and JIT behavior.

Every benchmark returns the built call graphs so the JIT cannot elide the work. No production code is touched.

Impact using my personal development machine and Zulu Java 17: the ensemble benchmark runs in ~5.8 s versus ~8.1 s for the end-to-end probe, removing the ~30% verification noise and landing almost exactly on the profile's predicted ~70% analysis portion. The per-phase methods attribute cost across algorithms (0-1-Container-CFA dominates at ~1.9 s), giving AB experiments a cheaper, lower-variance, call-graph-specific number to track optimizations.

Add JMH macro-benchmarks for the call-graph workload that measure only
pointer-analysis construction, not the test-harness verification the
existing end-to-end benchmark includes.

The dominant call-graph benchmark (`testHelloAllEntrypoints`) times a
whole unit test; a JFR profile showed ~30% of its wall time is spent
verifying the results (graph-integrity checks, call-graph squashing and
subset assertions, stat printing) rather than in the analysis itself.
That verification pollutes the timed signal with JIT'ed, IO-bound, and
assertion work, inflates the AB-experiment cycle, and obscures what real
WALA users pay for when building call graphs.

Split the five-phase pipeline in `CallGraphTest.doCallGraphs` into a
build step (all five call graphs, sharing one analysis cache) and the
existing verification step. The verification remains unchanged and
always runs in the test path, in the same phase order, so the
graph-structure and determinism assertions are exactly as before.

Add two kinds of benchmarks on top:

* per-phase single-shot benchmarks (RTA, 0-CFA, 0-1-CFA,
  0-Container-CFA, 0-1-Container-CFA) for cost attribution, with the
  scope, class hierarchy, options, and cache built once outside the
  timed region;

* an ensemble benchmark that runs all five phases in sequence with no
  verification, preserving the real pipeline's cross-phase cache and JIT
  behavior.

Every benchmark returns the built call graphs so the JIT cannot elide
the work. No production code is touched.

Impact using my personal development machine and Zulu Java 17: the
ensemble benchmark runs in ~5.8 s versus ~8.1 s for the end-to-end
probe, removing the ~30% verification noise and landing almost exactly
on the profile's predicted ~70% analysis portion. The per-phase methods
attribute cost across algorithms (0-1-Container-CFA dominates at
~1.9 s), giving AB experiments a cheaper, lower-variance,
call-graph-specific number to track optimizations.
@liblit
liblit requested a review from msridhar September 10, 2026 00:38
@liblit liblit self-assigned this Sep 10, 2026
@codecov

codecov Bot commented Sep 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.81%. Comparing base (36ae5ff) to head (971b959).

Additional details and impacted files
@@            Coverage Diff            @@
##             master    #2053   +/-   ##
=========================================
  Coverage     50.80%   50.81%           
  Complexity    12760    12760           
=========================================
  Files          1368     1368           
  Lines         83500    83500           
  Branches      14414    14414           
=========================================
+ Hits          42426    42427    +1     
  Misses        36388    36388           
+ Partials       4686     4685    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@liblit
liblit added this pull request to the merge queue Sep 10, 2026
Merged via the queue into wala:master with commit f58f4d0 Sep 10, 2026
11 checks passed
@liblit
liblit deleted the perf/callgraph-analysis-benchmarks branch September 10, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants