Many workflows failing with `EventCorruption`

[makortel]

Many workflows

• 136.90[1-6]
• 512.0 - 530.0
• 551.0, 552.0
• 554.0 - 556.0
• 562.0
• 1370.{0,17,18}
• 2500.910[22]
  are failing since CMSSW_20_1_X_2026-09-25-2300 with EventCorruption exception along

----- Begin Fatal Exception 26-Sep-2026 07:08:18 CEST-----------------------
An exception of category 'EventCorruption' occurred while
   [0] Processing global begin Run run: 1
   [1] Calling method for module Pythia8HadronizerFilter/'generator'
Exception Message:
No LHERunInfoProduct present
----- End Fatal Exception -------------------------------------------------

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el9_amd64_gcc14/CMSSW_20_1_X_2026-09-25-2300/pyRelValMatrixLogs/run/136.901_RunDoubleMuon2016C/step4_RunDoubleMuon2016C.log#/

In addition, 2500.9101 step 2 fails with FileNotFound

26-Sep-2026 07:40:05 CEST  Initiating request to open file file:step2_PAT.root
----- Begin Fatal Exception 26-Sep-2026 07:40:08 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling File::sysopen()
Exception Message:
Failed to open the file 'step2_PAT.root'
   Additional Info:
      [a] Attempted to open logical file file:step2_PAT.root.
      [b] Failed to open the file with physical name file:step2_PAT.root.
      [c] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

PRs merged in CMSSW_20_1_X_2026-09-25-2300

• Introducing new offline DQM sequences for HLT #51966
• MTD geometry: move MTD test default to D127 #51944
• Generate pileup minbias input by default in matrix workflows #51939

The #51966 and #51944 seem quite improbable cause to me.

[makortel]

Inspecting workflow 512.0 step 2, in 09-25-1100 the cmsDriver command is

cmsDriver.py Hadronizer_TuneCP5_13TeV_MLM_5f_max4j_LHE_pythia8_cff  --relval 200000,5000 -s GEN,VALIDATION:genvalid_all -n 10 --conditions auto:run2_mc --beamspot Realistic8TeVCollision --datatier GEN-SIM,DQMIO --eventcontent RAWSIM,DQM  --customise Validation/Performance/TimeMemorySummary.customiseWithTimeMemorySummary --prefix 'python monitor_workflow.py timeout --signal SIGTERM 9000 '  --filein  file:step1.root  --fileout file:step2.root  --suffix "-j JobReport2.xml "  --nThreads 4 > step2_DYTollJets_LO_Mad_13TeV_py8.log  2>&1

while in 09-25-2300 the command is

cmsDriver.py Hadronizer_TuneCP5_13TeV_MLM_5f_max4j_LHE_pythia8_cff  --relval 200000,5000 -s GEN,VALIDATION:genvalid_all -n 10 --conditions auto:run2_mc --beamspot Realistic8TeVCollision --datatier GEN-SIM,DQMIO --eventcontent RAWSIM,DQM  --customise Validation/Performance/TimeMemorySummary.customiseWithTimeMemorySummary --prefix 'python monitor_workflow.py timeout --signal SIGTERM 9000 '  --fileout file:step2.root  --suffix "-j JobReport2.xml "  --nThreads 4 > step2_DYTollJets_LO_Mad_13TeV_py8.log  2>&1

The latter is missing --filein file:step1.root

[makortel]

assign Configuration/PyReleaseValidation

[makortel]

@kpedro88

[cmsbuild]

New categories assigned: pdmv

@antoniovagnerini,@DickyChant,@kfjack,@miquork,@sroychow,@teteumac you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

In addition, 2500.9101 step 2 fails with FileNotFound

Similarly here, 09-25-1100 has

cmsDriver.py step2  -s NANO:@GEN,DQM:@nanogenDQM --process NANO --mc  --eventcontent NANOAODSIM,DQM --datatier NANOAODSIM,DQMIO -n 1000 --customise "Configuration/DataProcessing/Utils.addMonitoring" --conditions auto:run2_mc  --customise Validation/Performance/TimeMemorySummary.customiseWithTimeMemorySummary --prefix 'python monitor_workflow.py timeout --signal SIGTERM 9000 '  --filein  file:step1.root  --fileout file:step2.root  --suffix "-j JobReport2.xml "  --nThreads 4 > step2_DYToLL_M-50_13TeV_pythia8.log  2>&1

and 09-25-2300 has

cmsDriver.py step2  -s NANO:@GEN,DQM:@nanogenDQM --process NANO --mc  --eventcontent NANOAODSIM,DQM --datatier NANOAODSIM,DQMIO -n 1000 --customise "Configuration/DataProcessing/Utils.addMonitoring" --conditions auto:run2_mc  --customise Validation/Performance/TimeMemorySummary.customiseWithTimeMemorySummary --prefix 'python monitor_workflow.py timeout --signal SIGTERM 9000 '  --fileout file:step2.root  --suffix "-j JobReport2.xml "  --nThreads 4 > step2_DYToLL_M-50_13TeV_pythia8.log  2>&1

and the latter is missing --filein file:step1.root

[kpedro88]

Yes, it is possible one of the changes in #51939 had an unintended effect on non-upgrade workflows. Unfortunate that the PR tests provided no examples of such an affected workflow. I will propose a fix ASAP.

[makortel]

Maybe it would be worth of adding one LHE workflow to the limited matrix? @cms-sw/generators-l2 @cms-sw/pdmv-l2

Or in some other way extend the set workflows to cover all different kinds of workflows when a PR touches Configuration/PyReleaseValidation or Configuration/Applications?

[kpedro88]

I think #52004 should solve it.