Frequent failures in hlt P2 timing tests in IBs

[mmusich]

I noticed that since when we moved the hlt_p2_timing tests to use CMSSW_20_0_0_pre1 input files (PR cms-sw/cmssw#51216), there are frequent failures in IB (seemingly due to missing the input files), examples:

• CMSSW_20_1_X_2026-06-24-2300: log
• CMSSW_20_1_X_2026-06-26-1100: log
• CMSSW_20_1_X_2026-06-26-2300: log

Are the input files cached in the bot user area? If not, is it possible to do it?
Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @mmusich.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

@mmusich , I do not see any failures since Jun 29th. may be thing are stable now?

[mmusich]

I do not see any failures since Jun 29th. may be thing are stable now?

indeed there hasn't been any recently. Do you happen to know what changed?

[smuzaffar]

no idea, the data files are available since June 15th [a]. The IB tests for June 28th 23h00 were able to find the input files [b] while IB tests for June 29th 11h00 failed to get files from local disk [c]. The cmssw code for finding files just do ls of local directory, so no idea why it did not find the input files

[a]

> find  /data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1 -type f | xargs ls -l
-rw-r--r--. 1 cmsbuild zh 12559301370 Jun 15 16:06 /data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/0033230b-a131-453a-95c0-fe14d5027d1f.root
-rw-r--r--. 1 cmsbuild zh 12610075209 Jun 15 16:08 /data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/01131c24-728d-4ea0-850b-800a85023820.root
-rw-r--r--. 1 cmsbuild zh 12621344799 Jun 15 16:10 /data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/040e0617-27dc-487c-9f25-4002cae4d200.root
-rw-r--r--. 1 cmsbuild zh 12640196566 Jun 15 16:12 /data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/052197a8-b7fe-46a2-a0d1-9f52b1cb9d43.root

[b] https://cmssdt.cern.ch/SDT/jenkins-artifacts/hlt-p2-timing/CMSSW_20_1_X_2026-06-28-2300/el9_amd64_gcc13/run.log

Local repository: |/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000|
Discovered files: file:/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/0033230b-a131-453a-95c0-fe14d5027d1f.root,file:/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/01131c24-728d-4ea0-850b-800a85023820.root,file:/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/040e0617-27dc-487c-9f25-4002cae4d200.root,file:/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/052197a8-b7fe-46a2-a0d1-9f52b1cb9d43.root
L1P2GT,HLT:75e33_timing,ENDJOB
entry file:/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/0033230b-a131-453a-95c0-fe14d5027d1f.root
entry file:/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/01131c24-728d-4ea0-850b-800a85023820.root
entry file:/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/040e0617-27dc-487c-9f25-4002cae4d200.root
entry file:/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000/052197a8-b7fe-46a2-a0d1-9f52b1cb9d43.root

[c] https://cmssdt.cern.ch/SDT/jenkins-artifacts/hlt-p2-timing/CMSSW_20_1_X_2026-06-29-1100/el9_amd64_gcc13/run.log

Local repository: |/data/user/cmsbuild//store/relval/CMSSW_20_0_0_pre1/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_D121_RegeneratedGS_PU-v1/2590000|
Discovered files: 
L1P2GT,HLT:75e33_timing,ENDJOB
entry Phase2_RAW.root

[smuzaffar]

ah I noticed that failing jobs were running on host with T4 GPU [a] while the one which ran successfully were ran on host with L4 GPU

[a] https://cmssdt.cern.ch/SDT/jenkins-artifacts/hlt-p2-timing/CMSSW_20_1_X_2026-06-29-1100/el9_amd64_gcc13/run.log

Found 2 devices:
  - Tesla T4
  - Tesla T4

[b] https://cmssdt.cern.ch/SDT/jenkins-artifacts/hlt-p2-timing/CMSSW_20_1_X_2026-06-29-2300/el9_amd64_gcc13/run.log

2 visible NVIDIA CUDA GPUs:
  0: NVIDIA L4
  1: NVIDIA L4

[mmusich]

ah I noticed that failing jobs were running on host with T4 GPU [a] while the one which ran successfully were ran on host with L4 GPU

interesting! Why did the jobs end up running on the T4 host? They should never do that, because otherwise the comparison with the PRs will be not meaningful. Is it possible to actively prevent this to happen?

[smuzaffar]

ah ok, I found the reason. when this job is triggered via a PR test then it was not setting the correct parameters for the host selection e.g. https://cmssdt.cern.ch/jenkins/job/ib-run-hlt-p2-timing/3722/ was run on T4. Triggering of hlt timing job via PR tests were enabled when we were transitioning from T4 to L4 host. We can disable these now

[smuzaffar]

this is fixed now. HLT P2 timings jobs should always run on the HLT node configured via https://github.com/cms-sw/cms-bot/blob/master/ci-cd_config.sh#L1