Skip to content

[NGT] Introduce general DQM Validation for Secondary Vertexing - #51577

Merged
cmsbuild merged 26 commits into
cms-sw:masterfrom
JanGerritSchulz:jgs_SecondaryVertexValidation
Aug 12, 2026
Merged

[NGT] Introduce general DQM Validation for Secondary Vertexing#51577
cmsbuild merged 26 commits into
cms-sw:masterfrom
JanGerritSchulz:jgs_SecondaryVertexValidation

Conversation

@JanGerritSchulz

Copy link
Copy Markdown
Contributor

PR description:

This PR introduces a new DQM validation for Secondary Vertexing (SV). It follows a similar approach as primary vertex validation, and recycles+extends some of the existing infrastructure:

  • MC truth: TrackingVertices are already there and provide (almost) all the necessary information
  • Associatiors: Vertex association (Sim $\leftrightarrow$ Reco) is extended/templated to allow for all relevant types of vertices in CMSSW:
    • TrackingVertex $\leftrightarrow$ reco::Vertex already exists (same as for PV) $\to$ type of SVs built from RecoTracks directly
    • TrackingVertex $\leftrightarrow$ reco::VertexCompositePtrCandidate added by templating the class $\to$ type of SVs built from PF candidates (default secondary vertexing in HLT)
    • Extend the matching logic in the main associator used:
      • Make filtering to PV-only vertices optional via flag filterSimVerticesForPVs (enabled for PV, disabled for SV)
      • Add optional compatibility check for x/y positions (mainly for SV validation)
      • Enable associator to work with multiple track association maps (needed for SVs since PF candidates can refer to tracks from different collections)
      • Enable usage of number of shared tracks instead of just fractions for matching
  • DQM Analyzer: Completely new EDAnalyzer SecondaryVertexAnalyzerBase<VertexCollection> specifically for SV validation:
    • SecondaryVertexAnalyzer = SecondaryVertexAnalyzerBase<std::vector<reco::Vertex>>
    • SecondaryVertexAnalyzerCPC = SecondaryVertexAnalyzerBase<std::vector<reco::VertexCompositePtrCandidate>>
  • Set of DQM booking helpers: Addition of DQMBookingHelpers.h which serves as a centralized version of the helper functions already in use in the MultiTrackValidation (MTV) here. The respective ones there could technically be replaced in the future. If these helpers are not considered generally useful, this part of the PR is just a proposal and could easily be replaced by local helpers similar to the MTV.

More details in the slides linked at the top.

The new validation is also directly included in the central DQM validation workflow for HLT using the two specific sequences HLTSecondaryVertexValidation and HLTSecondaryVertexPostProcessorSequence. In the DQM output file, the added histograms are put in a new subfolder "DQMData/Run 1/HLT/Run summary/SecondaryVertices/Validation" by default. Example plots can be found in the slides.

PR validation:

I tested on larger scale using 9000 TTbar events with 200 PU to produce the plots in the presentation. To run a simple example on small scale, you can use this recipe:

#!/bin/bash –ex

cmsDriver.py step_2 -s L1P2GT,HLT:NGTScouting,VALIDATION:@hltValidation \
 --mc \
 --conditions auto:phase2_realistic_T35 \
 --datatier DQMIO \
 -n -1 \
 --eventcontent DQMIO \
 --geometry ExtendedRun4D121 \
 --era Phase2C17I13M9 \
 --customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_1000 \
 --filein file:/eos/cms/store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/0033230b-a131-453a-95c0-fe14d5027d1f.root \
 --fileout file:step2.root \
 --nThreads 24 \
 --process HLTX \
 --inputCommands='keep *, drop *_hlt*_*_HLT, drop triggerTriggerFilterObjectWithRefs_l1t*_*_HLT' \
 > step2.log 2>&1
 
cmsDriver.py step3 -s HARVESTING:@hltValidation \
 --conditions auto:phase2_realistic_T35 \
 --mc \
 --geometry ExtendedRun4D121 \
 --scenario pp \
 --filetype DQM \
 --era Phase2C17I13M9 \
 -n -1 \
 --filein file:step2.root \
 --fileout file:step3.root > step3.log 2>&1

I ran this with and without the PR changes applied. All existing validation histograms seem unchanged, specifically the primary vertexing ones. I think the usual bot tests should be sufficient to test a bit more extensively.

I also did a quick and dirty timing check using RelVal samples, and I couldn't see any significant differences in the timing pie. The SV analyzer itself takes in the order of 10-20 ms.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

This is not a backport. However, a backport could follow if there's interest to use it for Run-3 performance evaluation as well.

FYI @SWuchterl @philippgadow @pavlo-kashko @giovannicelotto

@JanGerritSchulz

Copy link
Copy Markdown
Contributor Author

type ngt

@cmsbuild

cmsbuild commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@cmsbuild

Copy link
Copy Markdown
Contributor

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51577/50346

Code check has found code style and quality issues which could be resolved by applying following patch(s)

@cmsbuild

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51577/50347

@cmsbuild

Copy link
Copy Markdown
Contributor

A new Pull Request was created by @JanGerritSchulz for master.

It involves the following packages:

  • DQMServices/Core (dqm)
  • DataFormats/Candidate (reconstruction)
  • DataFormats/VertexReco (reconstruction)
  • HLTriggerOffline/Common (dqm)
  • SimDataFormats/Associations (simulation)
  • SimTracker/TrackAssociation (simulation)
  • SimTracker/TrackHistory (simulation)
  • SimTracker/VertexAssociation (simulation)
  • Validation/RecoTrack (dqm)
  • Validation/RecoVertex (dqm)

@Moanwar, @civanch, @cmsbuild, @ctarricone, @gabrielmscampos, @jfernan2, @kpedro88, @mandrenguyen, @mdhildreth, @rseidita, @srimanob can you please review it and eventually sign? Thanks.
@AlexDeMoor, @GiacomoSguazzoni, @Ming-Yan, @Senphy, @VinInn, @VourMa, @abbiendi, @andrzejnovak, @apsallid, @barvic, @bsunanda, @castaned, @denizsun, @dgulhan, @elusian, @fabiocos, @felicepantaleo, @gpetruc, @jhgoh, @martinamalberti, @missirol, @mmasciov, @mmusich, @mtosi, @pavlo-kashko, @philippgadow, @rovere, @salimcerci, @threus, @venturia, @wmtford this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@mmusich

mmusich commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

test parameters:

  • workflows = ph2_hlt

@mmusich

mmusich commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Failed Tests: HeaderConsistency RelVals AddOn
Size: This PR adds an extra 208KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a96614/54968/summary.html
COMMIT: aa7ce12
CMSSW: CMSSW_20_1_X_2026-07-28-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51577/54968/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

----- Begin Fatal Exception 28-Jul-2026 19:38:00 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'prevalidation_step'
   [2] Calling method for module MultiTrackValidator/'trackValidatorFromPVAllTP'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::VertexToTrackingVertexAssociator<std::vector<reco::io_v1::Vertex> >
Looking for module label: VertexAssociatorByPositionAndTracks
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 28-Jul-2026 19:38:01 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'prevalidation_step'
   [2] Calling method for module MultiTrackValidator/'trackValidatorFromPVAllTP'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::VertexToTrackingVertexAssociator<std::vector<reco::io_v1::Vertex> >
Looking for module label: VertexAssociatorByPositionAndTracks
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 28-Jul-2026 19:39:48 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'prevalidation_step'
   [2] Calling method for module MultiTrackValidator/'trackValidatorFromPVAllTP'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::VertexToTrackingVertexAssociator<std::vector<reco::io_v1::Vertex> >
Looking for module label: VertexAssociatorByPositionAndTracks
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------
Expand to see more relval errors ...

Failed AddOn Tests

----- Begin Fatal Exception 28-Jul-2026 19:37:54 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 3
   [1] Running path 'prevalidation_step'
   [2] Calling method for module MultiTrackValidator/'trackValidatorFromPVAllTP'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::VertexToTrackingVertexAssociator<std::vector<reco::io_v1::Vertex> >
Looking for module label: VertexAssociatorByPositionAndTracks
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 28-Jul-2026 19:37:56 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 2 stream: 3
   [1] Running path 'prevalidation_step'
   [2] Calling method for module MultiTrackValidator/'trackValidatorFromPVAllTP'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::VertexToTrackingVertexAssociator<std::vector<reco::io_v1::Vertex> >
Looking for module label: VertexAssociatorByPositionAndTracks
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 28-Jul-2026 19:37:58 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 3
   [1] Running path 'prevalidation_step'
   [2] Calling method for module MultiTrackValidator/'trackValidatorFromPVAllTP'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::VertexToTrackingVertexAssociator<std::vector<reco::io_v1::Vertex> >
Looking for module label: VertexAssociatorByPositionAndTracks
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

@cmsbuild

cmsbuild commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Pull request #51577 was updated. @Moanwar, @civanch, @cmsbuild, @ctarricone, @gabrielmscampos, @jfernan2, @kpedro88, @mandrenguyen, @mdhildreth, @rseidita, @srimanob can you please check and sign again.

@mmusich

mmusich commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

- introduce flags ignoreMissingAssociations and ignoreMissingCollections
- if they are true LogWarnings are demoted to LogInfo
- if they are false and there are missing collections, the job will crash
- default setting is false
- preferred setting for HLT (where collections are expected to be missing) is true
@JanGerritSchulz
JanGerritSchulz force-pushed the jgs_SecondaryVertexValidation branch from 45d10c6 to 9f1262c Compare August 7, 2026 14:59
@JanGerritSchulz

Copy link
Copy Markdown
Contributor Author

Sorry, I just had to fix a typo in the validator cloning...

@cmsbuild

cmsbuild commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51577/50508

@cmsbuild

cmsbuild commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Pull request #51577 was updated. @Moanwar, @civanch, @cmsbuild, @ctarricone, @gabrielmscampos, @jfernan2, @kpedro88, @mandrenguyen, @mdhildreth, @rseidita, @srimanob can you please check and sign again.

@mmusich

mmusich commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

please test

@cmsbuild

cmsbuild commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

-1

Failed Tests: UnitTests
Size: This PR adds an extra 44KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a96614/55172/summary.html
COMMIT: 9f1262c
CMSSW: CMSSW_20_1_X_2026-08-07-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51577/55172/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test test_dqm-plot had ERRORS

Comparison Summary

Summary:

  • You potentially removed 25 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 61
  • DQMHistoTests: Total histograms compared: 4324977
  • DQMHistoTests: Total failures: 16
  • DQMHistoTests: Total nulls: 3
  • DQMHistoTests: Total successes: 4324940
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 31972.148000000005 KiB( 60 files compared)
  • DQMHistoSizes: changed ( 34434.7503,... ): 1598.605 KiB HLT/SecondaryVertices
  • DQMHistoSizes: changed ( 18434.0,... ): 0.004 KiB MessageLogger/Errors
  • DQMHistoSizes: changed ( 18434.0,... ): 0.004 KiB MessageLogger/Warnings
  • Checked 257 log files, 213 edm output root files, 61 DQM output files
  • TriggerResults: found differences in 20 / 60 workflows

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step3 max memory diff 32.3 exceeds +/- 30.0 MiB

@mmusich

mmusich commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

  • failure looks unrelated

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a96614/55217/summary.html
COMMIT: 9f1262c
CMSSW: CMSSW_20_1_X_2026-08-09-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51577/55217/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially removed 27 lines from the logs
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 61
  • DQMHistoTests: Total histograms compared: 4324977
  • DQMHistoTests: Total failures: 14
  • DQMHistoTests: Total nulls: 3
  • DQMHistoTests: Total successes: 4324942
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 31972.147999999994 KiB( 60 files compared)
  • DQMHistoSizes: changed ( 18434.0,... ): 1598.605 KiB HLT/SecondaryVertices
  • DQMHistoSizes: changed ( 18434.0,... ): 0.004 KiB MessageLogger/Errors
  • DQMHistoSizes: changed ( 18434.0,... ): 0.004 KiB MessageLogger/Warnings
  • Checked 257 log files, 213 edm output root files, 61 DQM output files
  • TriggerResults: found differences in 20 / 60 workflows

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step3 max memory diff 33.4 exceeds +/- 30.0 MiB

@gabrielmscampos

Copy link
Copy Markdown
Member

+dqm

@civanch

civanch commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

+1

1 similar comment
@Moanwar

Moanwar commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

+1

@cmsbuild

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

@mandrenguyen

Copy link
Copy Markdown
Contributor

+1

@perrotta

Copy link
Copy Markdown
Contributor

@JanGerritSchulz @mandrenguyen

After the merging of this PR, the following errors in the dictionaries are appearing during our builds (they were already visible in the "Duplicate Dictionaries" section of the test results of this PR):

./SimDataFormats/Associations/src/classes_def.xml defines the following dictionaries that should be defined in another package
  VertexReco : reco::VertexToTrackingVertexAssociator< vector< reco::Vertex > >
  VertexReco : edm::Wrapper< reco::VertexToTrackingVertexAssociator< vector< reco::Vertex > > >
  VertexReco : edm::helpers::KeyVal< edm::RefProd< vector< TrackingVertex > >, edm::RefToBaseProd< reco::Vertex > >
  VertexReco : edm::Wrapper< edm::helpers::KeyVal< edm::RefProd< vector< TrackingVertex > >, edm::RefToBaseProd< reco::Vertex > > >
  VertexReco : edm::helpers::KeyVal< edm::RefToBaseProd< reco::Vertex >, edm::RefProd< vector< TrackingVertex > > >
  VertexReco : edm::Wrapper< edm::helpers::KeyVal< edm::RefToBaseProd< reco::Vertex >, edm::RefProd< vector< TrackingVertex > > > >

Can they get fixed?

@JanGerritSchulz

Copy link
Copy Markdown
Contributor Author

Does the output mean that the dictionary definitions need to go under DataFormats/VertexReco instead?

For the first two, I am a bit puzzled though (but probably I am just not familiar enough with the setup in CMSSW). The reco::VertexToTrackingVertexAssociator lives in SimDataFormats/Associations/interface/VertexToTrackingVertexAssociator.h. So, I would have expected that this should be the correct package. Or is it because it is templated on reco::Vertex?

@perrotta

Copy link
Copy Markdown
Contributor

Does the output mean that the dictionary definitions need to go under DataFormats/VertexReco instead?

For the first two, I am a bit puzzled though (but probably I am just not familiar enough with the setup in CMSSW). The reco::VertexToTrackingVertexAssociator lives in SimDataFormats/Associations/interface/VertexToTrackingVertexAssociator.h. So, I would have expected that this should be the correct package. Or is it because it is templated on reco::Vertex?

Good point. I think @cms-sw/core-l2 can provide the correct answer and suggestions for it

@makortel

Copy link
Copy Markdown
Contributor

Yeah, the (not very smart) checker currently expects all class template instantiations with reco::Vertex template argument to be in DataFormats/VertexReco

{'VertexReco' : [f'reco::{io_v}Vertex']},

On a quick look the current placement of VertexToTrackingVertexAssociator is correct, and the checker needs to be updated. I'll take care of it.

@JanGerritSchulz

Copy link
Copy Markdown
Contributor Author

Okay, great. Thanks for the explanation 👍

@makortel

Copy link
Copy Markdown
Contributor

Done in #51700

Comment on lines +26 to +31
if (const auto *pkd = dynamic_cast<const pat::PackedCandidate *>(&cand)) {
(void)pkd; // suppress unused-variable warning pending the TODO above
return reco::TrackRef();
}

return reco::TrackRef();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to note that this effectively-no-op use of PackedCandidate resulted in #51709 . Is the TODO going to be addressed in the foreseeable future?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment. Reconsidering the point of the functions here, which is comparing vertices with TrackingVertices, and the fact that MiniAODs almost never even contain TrackingVertices (to my knowledge at least), I think the TODO is pointless anyway.

I would remove the TODO comment and the no-op if condition. Do you agree?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done here: #51719

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants