MC-truth graph prototype

Configuration/ProcessModifiers/python/enableTruth_cff.py

@@ -0,0 +1,7 @@
+# Original author: Felice Pantaleo (CERN) <felice.pantaleo@cern.ch>
+# Part of the MC-truth-graph prototype - under heavy development, not yet open
+# to external contributions (see PhysicsTools/TruthInfo/README.md).
+
+import FWCore.ParameterSet.Config as cms
+
+enableTruth = cms.Modifier()

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

@@ -1006,6 +1006,49 @@ def condition(self, fragment, stepList, key, hasHarvest):
 upgradeWFs['ticlv5_TrackLinkingGNN'].step3 = {'--procModifiers': 'ticlv5_TrackLinkingGNN'}
 upgradeWFs['ticlv5_TrackLinkingGNN'].step4 = {'--procModifiers': 'ticlv5_TrackLinkingGNN'}
 
+
+
+class UpgradeWorkflow_enableTruth(UpgradeWorkflow):
+    def setup_(self, step, stepName, stepDict, k, properties):
+        # enableTruth runs the truth-graph producers in RecoGlobal (step3) and,
+        # in GenSim (step1), keeps the full ancestor branch of every stored
+        # SimTrack (g4SimHits PersistencyEmin -> 0 via the modifier) so the
+        # truth graph stays connected to the generator. The Branch validators run
+        # in the RecoGlobal VALIDATION and their efficiency harvesting in
+        # HARVESTGlobal (step4), so the modifier must reach the harvesting step too.
+        if 'GenSim' in step or 'RecoGlobal' in step or 'HARVESTGlobal' in step:
+            stepDict[stepName][k] = deepcopy(stepDict[step][k])
+
+            if '--procModifiers' in stepDict[stepName][k]:
+                stepDict[stepName][k]['--procModifiers'] += ',enableTruth'
+            else:
+                stepDict[stepName][k]['--procModifiers'] = 'enableTruth'
+
+    def condition(self, fragment, stepList, key, hasHarvest):
+        return 'Run4' in key
+
+
+upgradeWFs['enableTruth'] = UpgradeWorkflow_enableTruth(
+    steps = [
+        'GenSim',
+        'GenSimHLBeamSpot',
+        'GenSimHLBeamSpot14',
+        'GenSimHLBeamSpotCloseBy',
+        'RecoGlobal',
+        'HARVESTGlobal',
+    ],
+    PU = [
+        'GenSim',
+        'GenSimHLBeamSpot',
+        'GenSimHLBeamSpot14',
+        'GenSimHLBeamSpotCloseBy',
+        'RecoGlobal',
+        'HARVESTGlobal',
+    ],
+    suffix = '_enableTruth',
+    offset = 0.88,
+)
+
 # L3 Tracker Muon Outside-In reconstruction first
 class UpgradeWorkflow_phase2L3MuonsOIFirst(UpgradeWorkflow):
     def setup_(self, step, stepName, stepDict, k, properties):

PhysicsTools/TruthInfo/BuildFile.xml

@@ -0,0 +1,14 @@
+<use name="FWCore/Framework"/>
+<use name="FWCore/MessageLogger"/>
+<use name="FWCore/Utilities"/>
+<use name="DataFormats/Common"/>
+<use name="DataFormats/Math"/>
+<use name="FWCore/ParameterSet"/>
+<use name="SimDataFormats/EncodedEventId"/>
+<use name="SimDataFormats/Associations"/>
+<use name="SimDataFormats/TruthInfo"/>
+<use name="heppdt"/>
+
+<export>
+  <lib name="1"/>
+</export>

PhysicsTools/TruthInfo/README.md

@@ -0,0 +1,90 @@
+# TruthInfo prototype
+
+> **Status: under heavy development — not open to external contributions.**
+> This is an experimental prototype: its data model, APIs, and configuration
+> change frequently and without notice, and it targets **Phase-2 (Run 4) only**
+> (no Phase-1/Run-2 support). Please do not submit external changes or depend on
+> it in production at this stage.
+>
+> **Original author and maintainer:** Felice Pantaleo (CERN),
+> <felice.pantaleo@cern.ch>.
+
+A prototype **MC-truth graph** for CMS: a single, navigable, physics-oriented
+abstraction of the generator + simulation truth history of an event, with
+calorimeter and tracker hit indices layered on top. It replaces the need to
+cross-navigate the many low-level truth collections (HepMC, GenParticles,
+SimTracks/SimVertices, TrackingParticles, SimClusters, CaloParticles, SimHits,
+RecHits) by hand.
+
+## Documentation
+
+The **authoritative, maintained documentation** lives on the project website:
+
+> **http://cms-truth.docs.cern.ch/**
+
+It covers the data model, how to enable and use the graph (navigation API, the
+`truth::Branch` view, the hit index, matching reco objects), the physics findings,
+validation, pileup, and the roadmap. The MkDocs sources are in the companion
+`cms-truth-docs` repository. Start with the **"How to use the graph"** page.
+
+## The three layers
+
+1. **`TruthGraph`** (raw) — a compact CSR graph built directly from HepMC +
+   `SimTrack`/`SimVertex` by `TruthGraphProducer`.
+2. **`truth::Graph`** (logical) — a user-facing bipartite Particle↔Vertex graph
+   built by `TruthLogicalGraphProducer`; GEN and SIM are merged where robustly
+   associated, with navigation (`parents()`, `descendants()`,
+   `firstCommonAncestor()`, `hasAncestorPdgId()`, …) and the `truth::Branch`
+   subgraph view + `BranchSelector` selection.
+3. **`truth::LogicalGraphHitIndex`** — per-particle direct vs aggregated subgraph
+   calorimeter and tracker hits, built by `LogicalGraphHitIndexProducer` (with the
+   DetId→RecHit map from `DetIdToRecHitMapProducer`).
+
+Producer chain (order matters): `truthGraphProducer` → `truthLogicalGraphProducer`
+→ `detIdToRecHitMapProducer` → `truthLogicalGraphHitIndexProducer`. In a release
+job these run behind the `enableTruth` process modifier (the
+`truthGraphPrevalidation` sequence in `Validation/Configuration`).
+
+## Package layout
+
+- `interface/` + `src/` — the data formats and algorithms: `TruthGraph`,
+  `truth::Graph`, `Branch`, `BranchSelector`, `LogicalGraphHitIndex`,
+  `BranchHitAssociator`, the `truth::recoHits` adapters (`RecoHitAdapters.h`), and
+  `TruthLogicalGraphPostProcessor` (merge/collapse/filter; covered by the cppunit).
+- `plugins/` — the producers above, the DOT dumpers, the flat-table producers, the
+  pileup `TruthGraphAccumulator`/`TruthGraphMixedProducer`, the association-map
+  producers (`TruthBranchCaloAssociationProducer`,
+  `TruthBranchTrackingAssociationProducer`), and the DQM validators
+  (`BranchHGCalValidator`, `BranchTrackingValidator`, the generic
+  `BranchRecoValidator`).
+- `python/` — `truthGraphValidation_cff` (producers + association maps + DQM
+  analyzers) and `truthGraphDQMHarvester_cff` (efficiency/fake/merge harvesting).
+- `scripts/` — `makeTruthGraphValidationPlots.py` (renders the Branch validation
+  plots / sample overlays).
+- `test/` — cppunit unit tests and standalone `cmsRun` drivers (graph dumps,
+  topology checks, association/DQM smoke tests).
+
+## Build, check, test
+
+All commands assume the CMSSW environment (`cd $CMSSW_BASE/src && cmsenv`):
+
+```bash
+scram b -j 8                          # build
+scram b code-format code-checks -j 8  # clang-format + clang-tidy (must pass)
+scram b runtests                      # cppunit unit tests
+```
+
+## Run standalone on a step3.root
+
+```bash
+# Dump per-event DOT graphs (options: -n, -m/--merge, -c/--collapse, -o, -t)
+cmsRun test/dumpTruthGraphsFromGENSIMRECO_cfg.py path/to/step3.root -n 5
+
+# Branch DQM validators (calo / tracking / generic reco-side)
+cmsRun test/validateBranchDQM_cfg.py          path/to/step3.root -n 5
+cmsRun test/validateBranchTrackingDQM_cfg.py  path/to/step3.root -n 5
+cmsRun test/validateBranchRecoDQM_cfg.py      path/to/step3.root -n 5
+```
+
+See the website for the full configuration reference, the navigation API with
+examples, and the validation results.

PhysicsTools/TruthInfo/doc/branch_design.md

@@ -0,0 +1,151 @@
+# `truth::Branch` — design proposal (for discussion)
+
+## Concept
+A `truth::Branch` is a **coherent connected subgraph** of `truth::Graph`: a chosen
+root (a particle, or a small set such as `Z -> mu mu`) together with a defined
+**closure** of its descendants, plus the detector footprint attached through
+`LogicalGraphHitIndex`. Like `Particle`/`Vertex`, it is a lightweight **view**
+into a `Graph`, not an owning copy.
+
+A Branch is the natural target for truth-reco association when a reconstructed
+object does not map to a single truth particle:
+- a jet  <-> a parton's whole shower branch,
+- an ECAL supercluster <-> a photon + its conversion e+e- branch,
+- a tau-jet <-> the visible tau-decay branch,
+- a b-jet  <-> the b-quark branch (with its B-hadron sub-branch).
+
+It is built directly on phases 1-3: the selection (seed PDG / heavy-flavor)
+chooses the root, the downstream closure defines the extent, the ISR/underlying
+-event roles + genEvent/eventId give provenance, and the hit index gives the
+detector footprint.
+
+## Construction & closure policy
+```cpp
+Branch b = graph.branch(rootParticle, closure);
+```
+`closure` selects which members belong to the branch and is always evaluated on
+the fly from the `Graph` (a Branch is never an EDM product):
+- `Subtree`     - the root and all descendants (default);
+- `StableLeaves`- the root plus only its final-state descendants;
+- `DepthN`      - descendants down to N generations;
+- `UntilPdgId`  - stop the closure at a species (e.g. stop at stable hadrons,
+                  or at the B hadron for a b-branch);
+- `Predicate`   - stop on a user predicate (stop at any heavy-flavor hadron, at a
+                  detector-boundary crossing, on a custom lambda), so closures
+                  are extensible without new enum values.
+The phase-1-3 postprocessing already computes member sets; a Branch makes that
+set a first-class, queryable object.
+
+**Decision:** the Branch is a **view, recomputed on demand** — stateless, no
+stored member list, not an EDM product. Any caching needed for performance lives
+in the *matching layer* (below), scoped to a batch of objects, not in the Branch.
+
+## Data model
+```cpp
+class Branch {
+  Graph const* graph_;
+  std::vector<uint32_t> roots_;      // usually 1
+  std::vector<uint32_t> members_;    // closure (materialized)
+  // optional caches: p4 sums, DetId set, hit spans
+};
+```
+A Branch carries provenance via its root (`genEvent`/`eventId`), so pile-up
+branches stay distinguishable when graphs are overlaid.
+
+## Queries the Branch should answer
+
+### A. Matching reco objects (the substrate is detector-agnostic; metrics are pluggable)
+- `members()`, `stableLeaves()`, `chargedStableLeaves()`.
+- `hits(closure)` - aggregated direct/subgraph SimHits + matched RecHits over all members (LogicalGraphHitIndex already gives this per particle).
+- `detIds()`, `energy(Detector)` - sim/rec energy summed over the branch in a subdetector.
+- `sharedHitFraction(recoObject)` / `sharedHits(recoObject)` - tracking-style metric.
+- `energyFraction(recoCluster)` - calorimeter-style metric.
+- `matchScore(recoObject, Metric)` - **Metric is a strategy on the Branch**: shared-hits (tracking), energy-fraction (calo), time-aware (MTD). New detectors add a Metric without touching Branch.
+- `containsSimTrack(id)`, `containsDetId(id)`.
+
+**Batch / many-to-many matching.** Single-object queries go through the Branch
+metric strategy above. For associating *collections* — N reco <-> 1 sim (split
+tracks, calo fragments -> one particle) and N sim <-> 1 reco (a jet <- a branch)
+— a free `BranchMatcher(branches, recoObjects, Metric)` builds the inverted
+`hit/DetId -> branch` index **once and caches it for the duration of the call**,
+then emits a weighted bipartite association in both directions (cf. reco's
+`RecoToSimCollection`/`SimToRecoCollection`). The Branch stays stateless; the
+cache lives in the matcher.
+
+**Hit ranges and `std::span`.** If `LogicalGraphHitIndex` lays hits out in graph
+-topological order, a `Subtree` branch's hits are a **contiguous range** — i.e.
+exactly the precomputed subgraph-hit `std::span` of its root, returned with zero
+gather. The matcher can then count shared hits / energy by a sorted-range
+merge-join rather than hashing, which is the cache-friendly path. (This needs the
+hit-index builder to guarantee the topological layout; see follow-up below.)
+
+### B. Tagging (flavor / origin / process)
+- `rootPdgId()`, `originPdgId(targets)` (= `firstAncestorWithPdgId` from the root).
+- `hasAncestorPdgId(id)` - is this branch from a top? a Z? the hard scatter?
+- `heavyFlavorContent()` - does the branch contain a b/c hadron (b/c-tag truth)? (reuses the phase-2 flavor classifier).
+- `isFromHardScatter()` / `isFromPileup()` - via the genEvent/eventId provenance (phase 1).
+- `decayChannel()` - decay mode of the root (Z->mumu vs Z->ee, tau hadronic vs leptonic, prong count).
+- `flightLength()` / `displacedVertex()` - production->decay displacement (Lxy) for b/tau lifetime tagging.
+
+### C. Physics performance
+- `p4(MemberSelector)` - branch four-momentum over {all | stable | charged | visible}.
+- `visibleEnergy()` / `invisibleEnergy()` - missing energy from neutrinos/LSP.
+- `chargedFraction()`, `emFraction()`, `hadronicFraction()` - for jet response/composition.
+- `response(recoObject)` = recoE / branchE; efficiency/fake bookkeeping via matching.
+- boundary-crossing kinematics from `Checkpoint`s - for propagation/calibration studies.
+
+### D. Branch <-> Branch relations
+- `commonAncestor(other)` - do two branches come from the same top / same Z? (generalizes `lowestCommonAncestor`).
+- `merged(other)` - combine two branches (and their hit content) into one (e.g. the two Z-decay legs).
+- `deltaR(other)`, `overlap(other)` (shared members/hits) - for splitting/merging studies.
+
+## Cross-cutting principles
+- **Substrate vs metric**: the Branch holds structure + hits; matching *definitions*
+  stay detector-aware and use-case dependent (tracking != calo != timing).
+- **Provenance-aware**: every Branch knows its source event (primary vs pile-up).
+- **Composable**: branches merge/split; queries compose with the existing
+  navigation (`ancestors`, `firstCommonAncestor`, `firstAncestorWithPdgId`).
+- **Built on what exists**: selection (phase 1-2), navigation (phase 3),
+  hit index (existing) — Branch is the unifying view, not new infrastructure.
+
+## Resolved decisions
+1. **View, recomputed on demand** — the Branch stores no member list and is not
+   an EDM product.
+2. **Derived on the fly** from `Graph` + closure (never persisted). Member-id
+   lists are cheap to recompute; hit aggregates are not stored on the Branch.
+3. **Metric strategy on the Branch** for single-object scores; a free,
+   cache-holding `BranchMatcher` for batch many-to-many association.
+4. **`merged()` unions member sets lazily** (no cached hit aggregates),
+   consistent with the view model.
+5. **Closures include predicate-based stops**, in addition to the fixed
+   `{Subtree, StableLeaves, DepthN, UntilPdgId}` set.
+
+## Implementation status
+Implemented (library level, all unit-tested):
+- **Hit layout** (`LogicalGraphHitIndex`): a particle's subgraph hits are a
+  contiguous, detId-sorted `std::span`, so a `Subtree` branch's hits are
+  `subgraphHits(root)` with zero gather and are merge-join ready.
+- **`truth::Branch`** (`interface/Branch.h`): the view, with closures
+  `Subtree / StableLeaves / DepthN / UntilPdgId / Predicate`, members/leaves,
+  p4 / visible / invisible energy, origin (`originWithPdgId`), heavy-flavor
+  content, pile-up provenance (`bunchCrossing`/`event`/`isSignal`/`isFromPileup`),
+  and relations (`commonAncestor`, `merged`).
+- **`truth::BranchHitAssociator`** (`interface/BranchHitAssociator.h`): the
+  generic, batch-cached matcher. **Customization point**: any reco object that
+  exposes `R::truthHits()` returning a range of `truth::RecoHit`
+  (`{detId, energy, fraction}`) is matchable — the `HasTruthHits<R>` concept. It
+  caches the inverted `detId -> roots` index once, then `bestBranches(reco)`
+  merge-joins the object's sorted hits against each candidate's sorted subgraph
+  span. Metrics: `SharedEnergy` (HGCal-style score) and `SharedHits`.
+- **`truth::BranchSelector`** (`interface/BranchSelector.h`): pt/eta/pdgId/charge
+  + signal/in-time selection, mirroring TrackingParticleSelector/CaloParticleSelector.
+
+## Remaining (EDProducer wiring)
+- Wrap `BranchHitAssociator` in EDProducers that consume real reco collections
+  (tracks, tracksters/PFclusters, jets) and emit `ticl::AssociationMap`
+  (`mapWithSharedEnergyAndScore`) products in both directions, mirroring
+  `AllTracksterToSimTracksterAssociatorsByHitsProducer` and the
+  TrackingParticle<->reco::Track associator, but with a Branch in place of the
+  SimTrackster/TrackingParticle.
+- A tracker variant keyed on shared `(trackId, EncodedEventId)` SimTrack hits
+  (the QuickTrackAssociatorByHits metric) for track<->branch matching.