Add a namespace to the NNPuppiTau model by thesps · Pull Request #49632 · cms-sw/cmssw

thesps · 2025-12-15T17:03:41Z

PR description:

We have had a recurring issue with hls4ml emulator models that are defined as externals that they sometimes clobber each other's symbols (arrays for weights typically) [1, 2]. This has been fixed by enforcing that new models define a namespace, with the other variables defined within. However we have one model that is defined in CMSSW source, whose development predates the externals setup, whose weights are not in a namespace, so those symbols are in the global namespace. This PR wraps that model in a namespace.

PR validation:

First of all checking symbols.
In the release:

$ readelf -Ws /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre6/lib/el9_amd
64_gcc12/pluginL1TriggerPhase2L1ParticleFlowAuto.so | grep w17
   728: 00000000001794f0    20 OBJECT  GLOBAL DEFAULT   25 w17
  4466: 00000000001794f0    20 OBJECT  GLOBAL DEFAULT   25 w17

In this PR:

$ readelf -Ws ../lib/el9_amd64_gcc12/pluginL1TriggerPhase2L1ParticleFlowAut
o.so | grep w17
  1363: 00000000001794f0    20 OBJECT  GLOBAL DEFAULT   25 _ZN20hls4ml_nnpuppitau_v03w17E
  4300: 00000000001794f0    20 OBJECT  GLOBAL DEFAULT   25 _ZN20hls4ml_nnpuppitau_v03w17E

I also checked the behaviour by defining a proxy external library that just defines and prints out some of these problematic symbols, and compiling it and loading it in CMSSW analagously to an hls4ml external. Working with the release, I see that the values are "successfuly" clobbered by the NNPuppiTau weights, while with the fix of this branch they retain their locally defined values. I ran a configuration file defined in FastPUPPI that ntuplizes the NNPuppiTau here just to run the producer in the job. There was no change in the ntuple produced since the model here is the one clobbering the others' weights.

cmsbuild · 2025-12-15T17:04:08Z

cms-bot internal usage

cmsbuild · 2025-12-15T17:06:09Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49632/47174

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49632/47174/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49632/47174/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

quinnanm · 2025-12-15T22:34:00Z

Thanks for this! Could you run code-format and update the PR?

cmsbuild · 2025-12-18T13:00:34Z

Milestone for this pull request has been moved to CMSSW_16_1_X. Please open a backport if it should also go in to CMSSW_16_0_X.

quinnanm · 2026-02-23T23:38:17Z

just following up on this: can you run code-format and update the PR when you get the chance? thanks

cmsbuild · 2026-04-04T12:40:50Z

Milestone for this pull request has been moved to CMSSW_17_0_X. Please open a backport if it should also go in to CMSSW_16_1_X.

cmsbuild · 2026-06-09T15:09:55Z

Pull request #49632 was updated. @BenjaminRS, @cmsbuild, @quinnanm can you please check and sign again.

BenjaminRS · 2026-06-09T15:10:56Z

test parameters:

workflow = 29634.78
relvals_opt = --what upgrade

BenjaminRS · 2026-06-09T15:11:21Z

please test

cmsbuild · 2026-06-09T16:24:36Z

-1

Failed Tests: RelVals
Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-545d5f/53778/summary.html
COMMIT: 568df19
CMSSW: CMSSW_20_1_X_2026-06-09-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49632/53778/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

34500.0

----- Begin Fatal Exception 09-Jun-2026 17:45:04 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'Phase2L1GTProducer'
   [2] Calling method for module L1GTProducer/'l1tGTProducer'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: std::vector<l1t::io_v1::PFTau>
Looking for module label: l1tNNTauProducerPuppi
Looking for productInstanceName: L1PFTausNN

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

34496.0

----- Begin Fatal Exception 09-Jun-2026 17:45:10 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'Phase2L1GTProducer'
   [2] Calling method for module L1GTProducer/'l1tGTProducer'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: std::vector<l1t::io_v1::PFTau>
Looking for module label: l1tNNTauProducerPuppi
Looking for productInstanceName: L1PFTausNN

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

34434.0

----- Begin Fatal Exception 09-Jun-2026 17:46:51 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'Phase2L1GTProducer'
   [2] Calling method for module L1GTProducer/'l1tGTProducer'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: std::vector<l1t::io_v1::PFTau>
Looking for module label: l1tNNTauProducerPuppi
Looking for productInstanceName: L1PFTausNN

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

quinnanm · 2026-06-11T23:36:40Z

+  void* handle = dlopen("libclobber.so", RTLD_NOW | RTLD_GLOBAL);
+  if (!handle) {
+    std::cerr << "dlopen failed: " << dlerror() << "\n";
+    return;
+  }
+
+  using f_t = void (*)();
+
+  dlerror();  // clear any old error
+  f_t f = reinterpret_cast<f_t>(dlsym(handle, "f"));
+
+  const char* err = dlerror();
+  if (err) {
+    std::cerr << "dlsym failed: " << err << "\n";
+    return;
+  }
+
+  f();                                                                // call into proxy


This seems to be causing the relval crashes- is this supposed to be here? maybe it's an unintended inclusion in the code-format commit?

this #include <dlfcn.h> is probably not intended as well? (link)

artlbv · 2026-06-18T10:50:11Z

would it be possible to move the NNTau code to using triggerflow @m-glowacki ?
that would save a lot of headache..

Wires the new cms-hls4ml/NNPuppiTauModel package into the SCRAM build following the L1TSC4NGJetModel/AXOL1TL precedent, so TauNNIdHW can load its weights via hls4mlEmulator::ModelLoader instead of in-tree global arrays (avoids the symbol-clobbering bug class in cms-sw/cmssw#49632).

Replaces the in-tree hls4ml weight arrays (w2/b2/.../w20/b20, global linkage, no namespace) with a call through hls4mlEmulator::Model, loaded at runtime via hls4mlEmulator::ModelLoader from the new cms-hls4ml/ NNPuppiTauModel external. This follows the same pattern already used for JetId/L1TSC4NGJetID, and avoids ELF symbol clashes when multiple hls4ml models are loaded into the same process (cms-sw#49632). - TauNNIdHW takes a std::shared_ptr<hls4mlEmulator::Model> in its constructor; EvaluateNN() becomes a thin prepare_input/predict/ read_result wrapper instead of inlining the dense/relu/sigmoid layer sequence. - L1NNTauProducer gains a ModelLoader member and a configurable tauModelPath parameter (default "NNPuppiTauModel_v1"), mirroring L1TSC4NGJetProducer. - Deletes the now-dead in-tree taus/{defines.h,tau_parameters.h,weights/, nnet_utils/} -- fully superseded by the external package. - Adds a CppUnit test (testNNPuppiTauModel.cppunit.cc) that loads the external model (including two instances at once, the scenario that used to clash on un-namespaced symbols) and sanity-checks predict() output.

Wires the new cms-hls4ml/NNPuppiTauModel package into the SCRAM build following the L1TSC4NGJetModel/AXOL1TL precedent, so TauNNIdHW can load its weights via hls4mlEmulator::ModelLoader instead of in-tree global arrays (avoids the symbol-clobbering bug class in cms-sw/cmssw#49632).

Mirrors the cms-hls4ml externals pattern used by L1TSC4NGJetModel/AXOL1TL: namespaces all hls4ml-generated weights, configs, and the dense/activation call sequence under hls4ml_nnpuppitaumodel_v1, and exposes only the extern "C" create_model()/destroy_model() factory pair across the .so boundary. This avoids ELF symbol clashes (w2/b2/.../w20/b20) when multiple hls4ml models are loaded into the same CMSSW process (cms-sw/cmssw#49632). NNPuppiTauModel_v1.{h,cpp} are a manual, logic-preserving extraction of TauNNIdHW::EvaluateNN()'s layer call sequence into a standalone namespaced free function, since this model's network body was never kept as a free function in CMSSW (unlike other hls4ml exports). Co-authored-by: artlbv <arturlbv@gmail.com>

Wires the new cms-hls4ml/NNPuppiTauModel package into the SCRAM build following the L1TSC4NGJetModel/AXOL1TL precedent, so TauNNIdHW can load its weights via hls4mlEmulator::ModelLoader instead of in-tree global arrays (avoids the symbol-clobbering bug class in cms-sw/cmssw#49632). Co-authored-by: artlbv <arturlbv@gmail.com>

Replaces the in-tree hls4ml weight arrays (w2/b2/.../w20/b20, global linkage, no namespace) with a call through hls4mlEmulator::Model, loaded at runtime via hls4mlEmulator::ModelLoader from the new cms-hls4ml/ NNPuppiTauModel external. This follows the same pattern already used for JetId/L1TSC4NGJetID, and avoids ELF symbol clashes when multiple hls4ml models are loaded into the same process (cms-sw#49632). - TauNNIdHW takes a std::shared_ptr<hls4mlEmulator::Model> in its constructor; EvaluateNN() becomes a thin prepare_input/predict/ read_result wrapper instead of inlining the dense/relu/sigmoid layer sequence. - L1NNTauProducer gains a ModelLoader member and a configurable tauModelPath parameter (default "NNPuppiTauModel_v1"), mirroring L1TSC4NGJetProducer. - Deletes the now-dead in-tree taus/{defines.h,tau_parameters.h,weights/, nnet_utils/} -- fully superseded by the external package. - Adds a CppUnit test (testNNPuppiTauModel.cppunit.cc) that loads the external model (including two instances at once, the scenario that used to clash on un-namespaced symbols) and sanity-checks predict() output. Co-authored-by: artlbv <arturlbv@gmail.com>

Mirrors the cms-hls4ml externals pattern used by L1TSC4NGJetModel/AXOL1TL: namespaces all hls4ml-generated weights, configs, and the dense/activation call sequence under hls4ml_nnpuppitaumodel_v1, and exposes only the extern "C" create_model()/destroy_model() factory pair across the .so boundary. This avoids ELF symbol clashes (w2/b2/.../w20/b20) when multiple hls4ml models are loaded into the same CMSSW process (cms-sw/cmssw#49632). NNPuppiTauModel_v1.{h,cpp} are a manual, logic-preserving extraction of TauNNIdHW::EvaluateNN()'s layer call sequence into a standalone namespaced free function, since this model's network body was never kept as a free function in CMSSW (unlike other hls4ml exports). Synced from artlbv/NNpuppitau.

Wires the new cms-hls4ml/NNPuppiTauModel package into the SCRAM build following the L1TSC4NGJetModel/AXOL1TL precedent, so TauNNIdHW can load its weights via hls4mlEmulator::ModelLoader instead of in-tree global arrays (avoids the symbol-clobbering bug class in cms-sw/cmssw#49632). Co-authored-by: artlbv <arturlbv@gmail.com>

Replaces the in-tree hls4ml weight arrays (w2/b2/.../w20/b20, global linkage, no namespace) with a call through hls4mlEmulator::Model, loaded at runtime via hls4mlEmulator::ModelLoader from the new cms-hls4ml/ NNPuppiTauModel external. This follows the same pattern already used for JetId/L1TSC4NGJetID, and avoids ELF symbol clashes when multiple hls4ml models are loaded into the same process (#49632). - TauNNIdHW takes a std::shared_ptr<hls4mlEmulator::Model> in its constructor; EvaluateNN() becomes a thin prepare_input/predict/ read_result wrapper instead of inlining the dense/relu/sigmoid layer sequence. - L1NNTauProducer gains a ModelLoader member and a configurable tauModelPath parameter (default "NNPuppiTauModel_v1"), mirroring L1TSC4NGJetProducer. - Deletes the now-dead in-tree taus/{defines.h,tau_parameters.h,weights/, nnet_utils/} -- fully superseded by the external package. - Adds a CppUnit test (testNNPuppiTauModel.cppunit.cc) that loads the external model (including two instances at once, the scenario that used to clash on un-namespaced symbols) and sanity-checks predict() output. Co-authored-by: artlbv <arturlbv@gmail.com>

Replaces the in-tree hls4ml weight arrays (w2/b2/.../w20/b20, global linkage, no namespace) with a call through hls4mlEmulator::Model, loaded at runtime via hls4mlEmulator::ModelLoader from the new cms-hls4ml/ NNPuppiTauModel external. This follows the same pattern already used for JetId/L1TSC4NGJetID, and avoids ELF symbol clashes when multiple hls4ml models are loaded into the same process (cms-sw#49632). - TauNNIdHW takes a std::shared_ptr<hls4mlEmulator::Model> in its constructor; EvaluateNN() becomes a thin prepare_input/predict/ read_result wrapper instead of inlining the dense/relu/sigmoid layer sequence. - L1NNTauProducer gains a ModelLoader member and a configurable tauModelPath parameter (default "NNPuppiTauModel_v1"), mirroring L1TSC4NGJetProducer. - Deletes the now-dead in-tree taus/{defines.h,tau_parameters.h,weights/, nnet_utils/} -- fully superseded by the external package. - Adds a CppUnit test (testNNPuppiTauModel.cppunit.cc) that loads the external model (including two instances at once, the scenario that used to clash on un-namespaced symbols) and sanity-checks predict() output. Co-authored-by: artlbv <arturlbv@gmail.com>

thesps · 2026-08-05T09:00:45Z

Superseded by #51263

Apply a namespace to the NNPuppiTau model

5984287

cmsbuild added this to the CMSSW_16_0_X milestone Dec 15, 2025

cmsbuild added l1-pending pending-signatures tests-pending orp-pending code-checks-pending labels Dec 15, 2025

cmsbuild added code-checks-rejected and removed code-checks-pending labels Dec 15, 2025

cmsbuild modified the milestones: CMSSW_16_0_X, CMSSW_16_1_X Dec 18, 2025

cmsbuild modified the milestones: CMSSW_16_1_X, CMSSW_17_0_X Apr 4, 2026

Apply code-format

461467c

cmsbuild added code-checks-pending and removed code-checks-rejected labels Apr 8, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 9, 2026

cmsbuild added tests-started and removed tests-pending labels Jun 9, 2026

cmsbuild added tests-rejected and removed tests-started labels Jun 9, 2026

quinnanm reviewed Jun 11, 2026

View reviewed changes

This was referenced Jun 19, 2026

Add NNPuppiTauModel external spec and toolfile cms-sw/cmsdist#10665

Merged

[Phase-2 L1T] Move TauNNIdHW to the externally-built NNPuppiTauModel package #51263

Merged

artlbv mentioned this pull request Jul 7, 2026

Move TauNNIdHW to the externally-built NNPuppiTauModel package artlbv/cmssw#3

Merged

thesps closed this Aug 5, 2026

Uh oh!

Conversation

thesps commented Dec 15, 2025

PR description:

PR validation:

Uh oh!

cmsbuild commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Dec 15, 2025

Uh oh!

quinnanm commented Dec 15, 2025

Uh oh!

cmsbuild commented Dec 18, 2025

Uh oh!

quinnanm commented Feb 23, 2026

Uh oh!

cmsbuild commented Apr 4, 2026

Uh oh!

cmsbuild commented Jun 9, 2026

Uh oh!

BenjaminRS commented Jun 9, 2026

Uh oh!

BenjaminRS commented Jun 9, 2026

Uh oh!

cmsbuild commented Jun 9, 2026

Failed RelVals

Uh oh!

quinnanm Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

quinnanm Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

artlbv commented Jun 18, 2026

Uh oh!

thesps commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cmsbuild commented Dec 15, 2025 •

edited

Loading