Addition of a HGCal-only reco, validation, harvesting sequence and workflow by silviodonato · Pull Request #51856 · cms-sw/cmssw

silviodonato · 2026-09-10T17:15:55Z

PR description:

In the context of NGT Task 3.3, where we are investigating the replacement of HGCal raw data with low-level reconstructed data, we found that CMSSW currently lacks an HGCal-only workflow that runs the HGCal reconstruction and DQM without the full reconstruction chain, analogous to the existing Pixel-, HCAL-, and ECAL-only workflows.

Such a workflow allows the HGCal reconstruction to be exercised very quickly by skipping the full general tracking.

This PR adds HGCal-only reconstruction, validation, and harvesting sequences, together with an upgrade workflow available through runTheMatrix.py:

runTheMatrix.py -w upgrade -l 37634.531 --extended --showMatrix

37634.531 TTbar_14TeV_TuneCP5_Run4D127_GenSimHLBeamSpot14+DigiTrigger_HGCalOnly_Run4D127+RecoGlobal_HGCalOnly_Run4D127+HARVESTGlobal_HGCalOnly_Run4D127 

[1]: cmsDriver.py TTbar_14TeV_TuneCP5_cfi  -s GEN,SIM -n 10 --conditions auto:phase2_realistic_T35 --beamspot DBrealisticHLLHC --datatier GEN-SIM --eventcontent FEVTDEBUG --geometry ExtendedRun4D127 --era Phase2C26I13M9 --relval 9000,100 

[2]: cmsDriver.py step2  -s DIGI:pdigi_valid,L1TrackTrigger,L1,L1P2GT,DIGI2RAW,HLT:@relvalRun4 --conditions auto:phase2_realistic_T35 --datatier GEN-SIM-DIGI-RAW -n 10 --eventcontent FEVTDEBUGHLT --geometry ExtendedRun4D127 --era Phase2C26I13M9

[3]: cmsDriver.py step3  -s RAW2DIGI:RawToDigi,RECO:reconstruction_hgcalOnly,VALIDATION:@HGCalValidationOnly --conditions auto:phase2_realistic_T35 --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO -n 10 --eventcontent FEVTDEBUGHLT,MINIAODSIM,DQM --geometry ExtendedRun4D127 --era Phase2C26I13M9 --customise Validation/Configuration/customiseHGCalOnly.customiseHGCalOnly

[4]: cmsDriver.py step4  -s HARVESTING:@HGCalValidationOnly --conditions auto:phase2_realistic_T35 --mc  --geometry ExtendedRun4D127 --scenario pp --filetype DQM --era Phase2C26I13M9 -n 100

Caveat:

Caveats:

particleFlowClusterHGCalNoTracks is configured with tracksterSrc = 'ticlTrackstersCLUE3DHigh', so that PF clustering uses the CLUE3D tracksters and does not require generalTracks.
The customiseHGCalOnly customisation sets trackingParticleGsfTrackAssociation.ignoremissingtrackcollection to True. This is needed because trackingParticleGsfTrackAssociation is still scheduled through the simulation-truth association dependency chain and otherwise attempts to consume the unavailable quickTrackAssociatorByHits product:

----- Begin Fatal Exception 10-Sep-2026 17:50:05 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 5
   [1] Running path 'validation_step'
   [2] Prefetching for module HGCalValidator/'hgcalValidator'
   [3] Prefetching for module SimTrackstersProducer/'ticlSimTracksters'
   [4] Calling method for module TrackAssociatorEDProducer/'trackingParticleGsfTrackAssociation'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::TrackToTrackingParticleAssociator
Looking for module label: quickTrackAssociatorByHits
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

I could use a Modifier to fix this, rather than a customization function, but it seems to me better to avoid adding a Modifier for a single workflow.

PR validation:

runTheMatrix.py -w upgrade -l 37634.531

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Yes, it would be useful to have it also in 20_0_X cycle, if possible, as it will be used for this kind of studies.

cmsbuild · 2026-09-10T17:16:27Z

cms-bot internal usage

cmsbuild · 2026-09-10T17:18:11Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51856/50891

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Rpc digi dev v9 #47447, Heterogeneous Alpaka implementation of electron pixel seed-matching #51200, Adding recHit masking to extended pixel tracks in heterogeneous CA #51268, Add ECAL Phase 2 development workflows with APD spikes #51636, Speed up runTheMatrix.py by up to 7x #51654, Update TenTau fragments for Phase-II workflows #51676, Add upgrade RelVal workflow for DYtoTauTautoTauLTauH #51748, [L1T] Phase2 L1Nano: drop hpsTauTable, and the dead HARVEST step from wf .781/.782 #51803, Add GEMs to FastSim #51810

mmusich · 2026-09-10T17:33:34Z

type ngt

mmusich · 2026-09-10T17:34:49Z

test parameters:

relval_opts = -w upgrade
workflow = 37634.531

mmusich · 2026-09-10T17:35:03Z

@cmsbuild, please test

mmusich · 2026-09-10T17:47:29Z

                   'standardValidationHiMix' : ['prevalidation','validationHiMix','validationHarvesting'],
                   'standardValidationNoHLTHiMix' : ['prevalidationNoHLT','validationNoHLTHiMix','validationHarvestingNoHLT'],
                   'HGCalValidation' : ['globalPrevalidationHGCal', 'globalValidationHGCal', 'hgcalValidatorPostProcessor'],
+                   'HGCalValidationOnly' : ['globalPrevalidationHGCal', 'globalValidationHGCalOnly', 'hgcalValidatorPostProcessor'],


May I suggest instead HGCalOnlyValidation ?

cmsbuild · 2026-09-10T19:33:24Z

-1

Failed Tests: RelVals RelVals-NVIDIA_L4
Size: This PR adds an extra 56KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3a18d7/55794/summary.html
COMMIT: 3f6ef76
CMSSW: CMSSW_20_1_X_2026-09-10-1100/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51856/55794/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

37634.7537634.75_TTbar_14TeV+Run4D127_HLT75e33Timing/step2_TTbar_14TeV+Run4D127_HLT75e33Timing.log

Failed RelVals-NVIDIA_L4

37634.750337634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid/step2_TTbar_14TeV+Run4D127_HLTHeterogeneousValid.log
37634.75137634.751_TTbar_14TeV+Run4D127_HLT75e33TimingAlpaka/step2_TTbar_14TeV+Run4D127_HLT75e33TimingAlpaka.log

cmsbuild · 2026-09-11T11:15:27Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51856/50898

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Rpc digi dev v9 #47447, Heterogeneous Alpaka implementation of electron pixel seed-matching #51200, Adding recHit masking to extended pixel tracks in heterogeneous CA #51268, Add ECAL Phase 2 development workflows with APD spikes #51636, Speed up runTheMatrix.py by up to 7x #51654, Update TenTau fragments for Phase-II workflows #51676, Add upgrade RelVal workflow for DYtoTauTautoTauLTauH #51748, [L1T] Phase2 L1Nano: drop hpsTauTable, and the dead HARVEST step from wf .781/.782 #51803, Add GEMs to FastSim #51810, Phase-2 primary-vertex reconstruction with the vertex-slot GNN (Alpaka inference), behind the vertexSlotGNN process modifier #51862

cmsbuild · 2026-09-11T11:15:49Z

Pull request #51856 was updated.

silviodonato · 2026-09-11T11:22:08Z

@cmsbuild, please test

cmsbuild · 2026-09-11T13:16:49Z

+1

Size: This PR adds an extra 52KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3a18d7/55804/summary.html
COMMIT: 55a79c8
CMSSW: CMSSW_20_1_X_2026-09-10-2300/el9_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51856/55804/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3592269
DQMHistoTests: Total failures: 24
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3592227
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files
TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 89 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 10011
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 163728
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 55 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 7912
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 165827
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 92 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 9773
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 163966
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_L4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 89 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 9195
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 164544
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 71 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 9649
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 164090
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 81 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 8623
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 165116
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

Max Memory Comparisons exceeding threshold NVIDIA_L4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff 190.6 exceeds +/- 30.0 MiB

rovere · 2026-09-11T14:31:05Z

What will happen to the output of the HGCAL reconstruction if it is run w/o any tracks in input?
Will it make still sense? Do we have charged hadrons in output?
Up to which level you would like this standalone HGCAL reconstruction to be "reliable" in terms of physics results?

silviodonato · 2026-09-11T17:01:49Z

cc @mmusich @younsig @cms-sw/hgcal-dpg-l2

What will happen to the output of the HGCAL reconstruction if it is run w/o any tracks in input?

I applied this change particleFlowClusterHGCalNoTracks.tracksterSrc = 'ticlTrackstersCLUE3DHigh' to run without tracks. I took the inspiration from this

Perhaps all objects are reconstruction as neutral, I don't know.

Do we have charged hadrons in output?

Maybe not, I don't know

Will it make still sense?

I think for testing it's still useful.
But we need to the full reco for a more precise test

Up to which level you would like this standalone HGCAL reconstruction to be "reliable" in terms of physics results?

@cms-sw/hgcal-dpg-l2 ?

cmsbuild · 2026-09-11T17:02:14Z

A new Pull Request was created by @silviodonato for master.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv)
Configuration/StandardSequences (operations)
Validation/Configuration (dqm, simulation)

@AdrianoDee, @DickyChant, @antoniovagnerini, @civanch, @ctarricone, @gabrielmscampos, @kfjack, @kpedro88, @mdhildreth, @miquork, @rseidita, @sroychow can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @Martin-Grunewald, @VinInn, @VourMa, @apsallid, @bdanzi, @denizsun, @dgulhan, @elusian, @fabiocos, @felicepantaleo, @makortel, @mmasciov, @mmusich, @mtosi, @rovere, @salimcerci, @sameasy, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

pfs · 2026-09-11T20:28:57Z

I think I need some context here. Is this aiming for a total failure scenario where tracker is absent and we can only produce photons and neutral hadrons as candidates? In any case @waredjeb @felicepantaleo please chime in, but I would imagine "hell breaking loose" with linking and trackster PID and regression algos currently trained assuming full CMS. I echo the questions from @rovere. Imho it would be nice to have a discussion on a TICL meeting before going ahead further.

cmsbuild added this to the CMSSW_20_1_X milestone Sep 10, 2026

cmsbuild added simulation-pending dqm-pending operations-pending pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending labels Sep 10, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 10, 2026

cmsbuild added the ngt label Sep 10, 2026

cmsbuild added tests-started and removed tests-pending labels Sep 10, 2026

mmusich reviewed Sep 10, 2026

View reviewed changes

cmsbuild added tests-rejected and removed tests-started labels Sep 10, 2026

cmsbuild mentioned this pull request Sep 11, 2026

Phase-2 primary-vertex reconstruction with the vertex-slot GNN (Alpaka inference), behind the vertexSlotGNN process modifier #51862

Open

add a wf and sequences to run HGCal-only reco, validation, harvesting

55a79c8

silviodonato force-pushed the HGCalOnlyReco_201X branch from 3f6ef76 to 55a79c8 Compare September 11, 2026 11:13

cmsbuild added tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels Sep 11, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 11, 2026

cmsbuild added tests-started and removed tests-pending labels Sep 11, 2026

cmsbuild added operations-approved tests-approved and removed operations-pending tests-started labels Sep 11, 2026

cmsbuild mentioned this pull request Sep 11, 2026

Heterogeneous Alpaka implementation of electron pixel seed-matching #51200

Open

silviodonato marked this pull request as ready for review September 11, 2026 16:52

Uh oh!

Conversation

silviodonato commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

Caveats:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

cmsbuild commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Sep 10, 2026

Uh oh!

mmusich commented Sep 10, 2026

Uh oh!

mmusich commented Sep 10, 2026

Uh oh!

mmusich commented Sep 10, 2026

Uh oh!

mmusich Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

silviodonato Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

cmsbuild commented Sep 10, 2026

Failed RelVals

Failed RelVals-NVIDIA_L4

Uh oh!

cmsbuild commented Sep 11, 2026

Uh oh!

cmsbuild commented Sep 11, 2026

Uh oh!

silviodonato commented Sep 11, 2026

Uh oh!

cmsbuild commented Sep 11, 2026

Comparison Summary

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L4 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Max Memory Comparisons exceeding threshold NVIDIA_L4

Uh oh!

rovere commented Sep 11, 2026

Uh oh!

silviodonato commented Sep 11, 2026

Uh oh!

cmsbuild commented Sep 11, 2026

Uh oh!

pfs commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

silviodonato commented Sep 10, 2026 •

edited

Loading

cmsbuild commented Sep 10, 2026 •

edited

Loading

pfs commented Sep 11, 2026 •

edited

Loading