MC-truth graph prototype by felicepantaleo · Pull Request #51213 · cms-sw/cmssw

felicepantaleo · 2026-06-13T21:28:50Z

PR description:

This PR introduces a prototype MC-truth graph for CMS: a single, navigable, event-level graph of the generator + simulation truth history, with calorimeter/tracker hit indices layered on top. The goal is to let reconstruction and validation code reason about truth in terms of stable physics abstractions (particles, vertices, decay branches, hits) instead of depending directly on the storage details of GenParticle, SimTrack, GenVertex, SimVertex, PCaloHit/PSimHit, and the various legacy truth objects (TrackingParticle, CaloParticle, SimCluster).

It is Phase-2 only at this stage. The authoritative design notes live in PhysicsTools/TruthInfo/README.md.

More information
https://cms-truth.docs.cern.ch/

Three-layer data model (all in PhysicsTools/TruthInfo):

TruthGraph (raw) — a compact CSR graph built directly from HepMC2/HepMC3 + SimTrack/SimVertex by TruthGraphProducer. Nodes are GenEvent/GenVertex/GenParticle/SimVertex/SimTrack; cross-domain GenToSim edges link the two realms. GEN↔SIM provenance is taken from the faithful per-track generator barcode (SimTrack::genpartIndex() for primary/pre-assigned tracks).
truth::Graph (logical) — a user-facing bipartite Particle↔Vertex graph built by TruthLogicalGraphProducer. GEN and SIM particles/vertices are merged when robustly associated; a merged particle takes its production vertex from its immediate GEN production vertex. Intermediate GEN-only copies can be collapsed. Navigation API: parents(), children(), ancestors(), descendants(), firstCommonAncestor(), lowestCommonAncestor(), hasAncestorPdgId(), …
truth::LogicalGraphHitIndex — a per-logical-particle calorimeter+tracker hit index (direct hits vs. aggregated subgraph hits, CSR/std::span-based), built by LogicalGraphHitIndexProducer from PCaloHit/PSimHit plus the DetId→RecHit-index map from a new SimHitToRecHitMapProducer.

On top of the graph:

truth::Branch — a recomputed-on-demand subgraph/decay-branch view with configurable closures (subtree, stable-leaves, depth-N, until-pdgId, arbitrary predicate), kinematics (p4/visibleP4/invisible energy), tagging (heavy-flavor content, origin-with-pdgId), pile-up/in-time/signal provenance, and relations (common ancestor, merge).
truth::BranchHitAssociator — generic hit-based reco↔truth matching that works on any reco object exposing a truthHits() method (a C++20 concept). It builds the same shared-energy/score metric used by the HGCal by-hits associator, plus a shared-hits metric, over both the calorimeter and tracker hit channels.
truth::BranchSelector — kinematic / pdgId / charge / signal-only selection, in the spirit of CaloParticleSelector / TrackingParticleSelector.
Auxiliary plugins: DOT dumpers (TruthGraphDumper, TruthLogicalGraphDumper) and NanoAOD-style flat-table producers.
A new enableTruth process modifier (Configuration/ProcessModifiers).
A truthPrevalidation sequence (Validation/Configuration), the producer chain truthGraphProducer → truthLogicalGraphProducer → simHitToRecHitMapProducer → truthLogicalGraphHitIndexProducer, hooked into globalValidation_cff behind enableTruth.
Under enableTruth, g4SimHits.TrackingAction.PersistencyEmin = 0 so every stored SimTrack keeps its full ancestor branch and the truth graph is fully connected to the generator.
A .88 upgrade-workflow variant (upgradeWorkflowComponents.py) that appends --procModifiers enableTruth to the GenSim and RecoGlobal steps of Run4 workflows, for relval-based validation.

New: PhysicsTools/TruthInfo (the package), SimCalorimetry/HGCalAssociatorProducers/{SimHitToRecHitMapProducer, DetIdRecHitMap} (not HGCal-specific), the enableTruth_cff modifier, and truthPrevalidation_cff. The only edits to existing files are the three modifier-gated hooks above (globalValidation_cff.py, g4SimHits_cfi.py, upgradeWorkflowComponents.py); with enableTruth off there is no change to any standard workflow.

PR validation:

Unit tests (cppunit, scram b runtests): 31 assertions across 5 binaries (TruthLogicalGraphPostProcessor_t, LogicalGraphHitIndexBuilder_t, Branch_t, BranchHitAssociator_t, BranchSelector_t). All pass.
Eight enableTruth Run4 D120 (no-PU) relvals run end-to-end (all steps PASSED): 34002.88 SingleElectronPt35, 34034.88 TTbar 14TeV, 34044.88 DYToLL M-50, 34045.88 DYToTauTau M-50, 34050.88 ZMM, 34052.88 H125 ggF, 34131.88 VBFHZZ4Nu, 34087.88 TenTau.
Topology audit over all eight samples: the logical graph is a proper DAG (no cycles), one connected component per event (no orphans), and physical vertex multiplicities (no artificial high-degree vertices).
Branch-vs-legacy cross-checks via the existing associators: a truth::Branch reproduces the hit content of CaloParticle/SimCluster (hit & energy completeness = 1.0; purity ~0.7–0.85, the Branch being the broader unified object) and of TrackingParticle through ClusterTPAssociation (track→truth agreement 96–100%).
Built and validated with identical behavior on CMSSW_17_0_0_pre2 and CMSSW_20_0_0_pre1.

some visual examples
https://felice.web.cern.ch/truth/

Notes for reviewers

This is an early WIP shared for design discussion; the data model and APIs are still expected to evolve, and the producers run only in validation behind enableTruth.

Store packed reco::GenStatusFlags in the raw TruthGraph and propagate them to the logical truth::Graph particle payload. Decode and print the active status flags in the logical graph dumper to make hard-process and copy-history information visible in DOT dumps.

Add an optional post-processing pass for the logical truth graph. The pass collapses GEN-only chains of the form P -> V -> C when P has status different from 1, C is the only daughter, and P and C have the same PDG id. This removes intermediate generator-history copies while preserving the physics-forward graph topology.

* Dump information also as DOT file attributes * Dump type= code format

Quote non-trivial custom DOT attributes emitted by TruthGraphDumper, including four-vectors, positions, node type strings, and boundary information. This avoids Graphviz parse errors from parentheses, commas, spaces, and angle-bracket syntax in generated DOT files.

Route logical truth graph cleanups through a dedicated postProcessGraph() function and a post-processing configuration object. The existing same-PDG intermediate GEN particle collapse is kept as the first post-processing pass, making it easier to add future graph filters and treatments without cluttering the producer control flow.

Avoid overwriting the nominal GEN four-momentum with the SimTrack momentum when building logical particles that merge GEN and SIM information. SimTrack momentum is now used only for SIM-only logical particles, while boundary information remains stored as checkpoints. Also clean up the logical graph producer post-processing flow.

code format

theofil · 2026-06-30T17:44:22Z

+1

civanch · 2026-07-04T11:52:36Z

@felicepantaleo , concerning MC truth at SIM step: we postpone modification of SimTrackManager for 20_1_X, because afraid destroy Run2,3 MC truth management. The current mechanism cannot work in asynchronous mode because inside SimTrackManager assumed strong order of particles. Because it is not the last modification of new MCtruth schema I am not sure what is the correct order of modifications. In my mind Phase2SimTrackManager class should be introduced together with corresponding modifications inside Phase2TrackingAction. I would also prefer if TrackingAction and SImTackManager stay unchanged, so for some time inside 20_1 we may perform Run3 type simulation in the same way as in 17_0_X.

Would this work for your schema?

Would your MC truth method survive if you drop from this PR modifications in SimTrackManager and TrackingAction?

The flags were zeroed pending a barcode-to-reco::GenParticle association, but no association is needed: MCTruthHelper computes reco::GenStatusFlags directly from the HepMC record (it is what GenParticleProducer uses). HepMC2 path only; the HepMC3 path stays 0 until MCTruthHelper grows a HepMC3 specialization. With this, isHardProcess and friends are available on the graph nodes; on VBF the two hard-scatter quark legs now carry kIsHardProcess (2.03 partonic top-level nodes per event in the branch SimTracksters, u/d dominated, no gluon-shower leakage).

cmsbuild · 2026-07-08T12:24:11Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51213/50125

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Rpc digi dev v9 #47447, Displaced Particle Gun #51006, Adding recHit masking to extended pixel tracks in heterogeneous CA #51268, Add additional processes to ECAL development matrix workflows. #51269, Add premixing to development ECAL Phase 2 simulation #51358
- File Validation/Configuration/python/globalValidation_cff.py modified in PR(s): Add ECAL Phase 2 digis and TP digis validation #51400

cmsbuild · 2026-07-08T12:24:38Z

Pull request #51213 was updated. @AdrianoDee, @DickyChant, @antoniovagnerini, @civanch, @cmsbuild, @ctarricone, @davidlange6, @fabiocos, @ftenchini, @gabrielmscampos, @kfjack, @kpedro88, @lviliani, @mandrenguyen, @mdhildreth, @miquork, @mkirsano, @rseidita, @sensrcn, @sroychow, @theofil can you please check and sign again.

felicepantaleo · 2026-07-08T12:31:40Z

@cmsbuild please test

felicepantaleo · 2026-07-08T13:33:50Z

@felicepantaleo , concerning MC truth at SIM step: we postpone modification of SimTrackManager for 20_1_X, because afraid destroy Run2,3 MC truth management. The current mechanism cannot work in asynchronous mode because inside SimTrackManager assumed strong order of particles. Because it is not the last modification of new MCtruth schema I am not sure what is the correct order of modifications. In my mind Phase2SimTrackManager class should be introduced together with corresponding modifications inside Phase2TrackingAction. I would also prefer if TrackingAction and SImTackManager stay unchanged, so for some time inside 20_1 we may perform Run3 type simulation in the same way as in 17_0_X.

Would this work for your schema?

Would your MC truth method survive if you drop from this PR modifications in SimTrackManager and TrackingAction?

I checked what dropping the SimTrackManager and TrackingAction modifications would do to the schema, with numbers.
Measured on 10 TTbar D121 events with the reconnection disabled (i.e. the unchanged code path): 28.5% of the stored SimTracks (15079 of 52845) and 32.1% of the calo-boundary-crossing ones (14836 of 46173) are disconnected from the generator: their ancestry ends at a parent dropped by PersistencyEmin (50 GeV), leaving SimVertex::parentIndex = -1. With the reconnection enabled (10 VBF H->ZZ->4nu D120 events) both numbers are exactly 0. The calo-boundary-crossing tracks are what the calorimeter truth is anchored to, so losing generator ancestry (status flags, decay-level grouping, prompt/secondary classification) for a third of them is a showstopper for the schema, not a graceful degradation.

The alternative that keeps SimTrackManager and TrackingAction strictly untouched is PersistencyEmin = 0 under the enableTruth modifier: same full connectivity, pure configuration change, but roughly 2.9x the stored SimTracks. The reconnection stores exactly the same track set as today and only fills the parent redirects, which is why I preferred it (about 65% fewer SimTracks than PersistencyEmin = 0 at identical connectivity).

On the asynchronous concern: the redirect does not assume any ordering of particles. It is computed after tracking as a pure walk of the complete trackID to parentID map; computeDroppedAncestorRedirect is a static function of that map alone, covered by a unit test in the PR (SimTrackManagerReconnect_t). And the change is behaviorally inert by default.

cmsbuild · 2026-07-08T14:36:55Z

+1

Size: This PR adds an extra 32KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f52ba8/54607/summary.html
COMMIT: 050607e
CMSSW: CMSSW_20_1_X_2026-07-07-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51213/54607/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3547841
DQMHistoTests: Total failures: 21
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3547802
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files
TriggerResults: no differences found

Moanwar · 2026-07-09T10:57:32Z

urgent

Moanwar · 2026-07-09T10:59:35Z

@cms-sw/simulation-l2 @cms-sw/generators-l2 @cms-sw/pdmv-l2 @cms-sw/dqm-l2,
Could you please review this PR so that we can merge it in time and have it backported for the upcoming Phase-2 sample production? Thanks!

civanch · 2026-07-09T12:16:37Z

+1

@felicepantaleo , I still am not sure that your statement is correct. Let us assume that in Run1,Run2,Run3 analyses 30% of useful track is lost in MC truth. In that case likely impossible make decent comparisons between MC and the data. I suspect, that what is lost is not important.

However, as we discussed at SIM meeting, let things go, so I sign the PR as it is.

vlimant · 2026-07-10T10:35:39Z

can one please point me to the place where this changes the GEN-SIM event content or or what are the data format changes which affect the GEN-SIM event content that goes in central production ?

felicepantaleo · 2026-07-10T11:25:16Z

can one please point me to the place where this changes the GEN-SIM event content or or what are the data format changes which affect the GEN-SIM event content that goes in central production ?

Hi @vlimant , everything is behind a process modifier, so no change is expected in central production.

I have ready a follow-up PR for the pileup mixing treatment, and to make it usable in central production adding under a MB/event to the event content in ttbar PU200

felicepantaleo · 2026-07-10T12:22:50Z

felicepantaleo/cmssw@truthGraph_CMSSW_20...truthGraph_pileup_production_CMSSW_20
Let me know if you want me to push it here.

Moanwar · 2026-07-10T12:40:48Z

can one please point me to the place where this changes the GEN-SIM event content or or what are the data format changes which affect the GEN-SIM event content that goes in central production ?

Well, @felicepantaleo can correct me if I'm wrong, but if we want to use this through a process modifier in the central production, it still needs to run from the GEN-SIM step. That means it needs to be included in CMSSW_20_0, which will be used for the production.

lviliani · 2026-07-10T12:44:17Z

+generators

gabrielmscampos · 2026-07-10T13:07:17Z

+dqm

The Validation/ bit is ok

felicepantaleo · 2026-07-10T16:34:43Z

can one please point me to the place where this changes the GEN-SIM event content or or what are the data format changes which affect the GEN-SIM event content that goes in central production ?

Well, @felicepantaleo can correct me if I'm wrong, but if we want to use this through a process modifier in the central production, it still needs to run from the GEN-SIM step. That means it needs to be included in CMSSW_20_0, which will be used for the production.

Correct. Sorry, I have misunderstood @vlimant question
A/B/C on the same generator seed, 10 events, CMSSW_20_1_X, 8 threads. Three configurations:

config	GEN-SIM	DIGI (PU200)
A baseline	standard	standard, no truth
B calo-only	`enableTruth`	+ `TruthGraphAccumulator` (HGCAL + barrel ECAL/HCAL)
C full-detector	`enableTruth`	+ accumulator with `includeTrackingHits=True` (adds tracker, muon, MTD)

Storage is exact; wall-clock times are indicative (the host had concurrent load, so use the deltas and the load-independent metrics).

GEN-SIM: `enableTruth` vs standard

enableTruth sets TrackingAction.ReconnectDroppedAncestors = True. The cost is CPU, not size:

	CPU (`time_thread`)	step1 size
standard	3.35 s/ev	2.37 MB/ev
`enableTruth`	3.98 s/ev (+19%)	2.37 MB/ev (+0%)

If you are interested also in DIGI-RAW storage:

DIGI-RAW storage (exact)

config	step2 size	vs baseline
A baseline	108.0 MB/ev	-
B calo-only	135.7 MB/ev	+27.7 (+26%)
C full-detector	157.9 MB/ev	+49.9 (+46%)

Truth-product breakdown (compressed, MB/event):

product	calo-only	full-detector
mergedTrackerHits	-	21.1
mergedEcalHits	15.6	14.8
mergedHGCHits	6.4	6.2
mergedMtdHits	-	4.2
TruthGraph	3.3	3.3
mergedHcalHits	0.3	0.3
mergedMuonHits	-	0.04

DIGI timing (MixingModule, where the accumulator runs)

config	MixingModule real	vs baseline
baseline	34.4 s/ev	-
calo-only	42.1 s/ev	+7.7 (+22%)
full-detector	54.8 s/ev	+20.5 (+60%)

Full chain (GEN-SIM + DIGI, per event)

config	storage	time_real
baseline	110 MB/ev	87 s/ev
calo-only	138 MB/ev (+25%)	97 s/ev (+11%)
full-detector	160 MB/ev (+45%)	110 s/ev (+26%)

felicepantaleo · 2026-07-10T19:40:30Z

@cms-sw/pdmv-l2 could you please sign svp?

AdrianoDee · 2026-07-11T21:18:41Z

test parameters:

relvals = 34034.88, 34050.88, 34087.88
relval_opts = -w upgrade

AdrianoDee · 2026-07-11T21:19:07Z

please test
(new wfs! it's worth to test them)

AdrianoDee · 2026-07-11T21:32:09Z

+pdmv

as far as the new tests run fine, changes are good wfs-wise.

cmsbuild · 2026-07-11T23:19:35Z

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f52ba8/54694/summary.html
COMMIT: 050607e
CMSSW: CMSSW_20_1_X_2026-07-11-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51213/54694/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3548186
DQMHistoTests: Total failures: 56
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3548112
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files
TriggerResults: no differences found

cmsbuild · 2026-07-11T23:19:38Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @ftenchini, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2026-07-12T08:50:32Z

+1

Moanwar · 2026-07-12T11:56:14Z

Hi @felicepantaleo, I believe this PR still needs to be backported to 20_0, no?

slava77 · 2026-07-21T14:44:24Z

A/B/C on the same generator seed, 10 events, CMSSW_20_1_X, 8 threads. Three configurations:

how did the CPU memory behave? (although it's unclear if 10 events across 8 threads (was it 8 streams?) are representative for a memory peak measurement)

felicepantaleo and others added 30 commits June 13, 2026 22:36

Introduce compact truth graph data structure

c0899fa

Add necessary build and dictionary files for TruthGraph

c8ec262

Implement TruthGraphProducer for event graph construction

0d0c995

Add dumper module for graph visualization and inspection

cff8453

Add plugin build configuration for producer and dumper modules

31ce40f

Create standalone truth logical graph

a48dd47

Draft README

49d1207

Add possibility to filter mother by pdgid

fbc4f71

Dump information also as DOT file attributes

8ca07ac

* Dump information also as DOT file attributes * Dump type= code format

Add extra DOT file labels to TruthLogicalGraphDumper (#64)

ced60b7

format

d4441c5

Process multiple files at same time

4a54f19

Build GEN-SIM particle links only for primary SimTracks

91e8d6d

Improve TruthLogicalGraph DOT dumping

5e74161

Label TruthGraph DOT edges with explicit edge types

55d8d42

Dot simToGen edge

a393d5e

Add SimHits to SimTracks

749908a

code format

Add new SimHitToRecHitMap

aadec46

Add associated rechits and energy to truth particles

979e429

Update README

4b29150

adding rechits

c8629b2

adding possibility to hide truth particles with no simhits

76f2963

add argparser

0429b70

Separate postprocessing

6cb9983

code format

make input file optional (#66)

180b280

cmsbuild mentioned this pull request Jul 3, 2026

Add ECAL Phase 2 digis and TP digis validation #51400

Merged

felicepantaleo mentioned this pull request Jul 12, 2026

20_0_X backport - MC-truth graph prototype #51469

Merged

cmsbuild mentioned this pull request Jul 12, 2026

TruthInfo: pileup-aware truth graph during mixing #51470

Merged

Uh oh!

Conversation

felicepantaleo commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

Notes for reviewers

Uh oh!

theofil commented Jun 30, 2026

Uh oh!

civanch commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jul 8, 2026

Uh oh!

cmsbuild commented Jul 8, 2026

Uh oh!

felicepantaleo commented Jul 8, 2026

Uh oh!

felicepantaleo commented Jul 8, 2026

Uh oh!

cmsbuild commented Jul 8, 2026

Comparison Summary

Uh oh!

Moanwar commented Jul 9, 2026

Uh oh!

Moanwar commented Jul 9, 2026

Uh oh!

civanch commented Jul 9, 2026

Uh oh!

vlimant commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felicepantaleo commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felicepantaleo commented Jul 10, 2026

Uh oh!

Moanwar commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lviliani commented Jul 10, 2026

Uh oh!

gabrielmscampos commented Jul 10, 2026

Uh oh!

felicepantaleo commented Jul 10, 2026

GEN-SIM: enableTruth vs standard

DIGI-RAW storage (exact)

DIGI timing (MixingModule, where the accumulator runs)

Full chain (GEN-SIM + DIGI, per event)

Uh oh!

felicepantaleo commented Jul 10, 2026

Uh oh!

AdrianoDee commented Jul 11, 2026

Uh oh!

AdrianoDee commented Jul 11, 2026

Uh oh!

AdrianoDee commented Jul 11, 2026

Uh oh!

cmsbuild commented Jul 11, 2026

Comparison Summary

Uh oh!

cmsbuild commented Jul 11, 2026

Uh oh!

mandrenguyen commented Jul 12, 2026

Uh oh!

Moanwar commented Jul 12, 2026

Uh oh!

slava77 commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

felicepantaleo commented Jun 13, 2026 •

edited

Loading

civanch commented Jul 4, 2026 •

edited

Loading

vlimant commented Jul 10, 2026 •

edited

Loading

felicepantaleo commented Jul 10, 2026 •

edited

Loading

Moanwar commented Jul 10, 2026 •

edited

Loading

GEN-SIM: `enableTruth` vs standard