Conversation
.gitignore covered *~ but not the swap files an editor leaves beside a file while it is open, which is how one of them ended up committed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WP5x6r6bsRbe8kEQEUJk1S
A copy of src/fwtest under src/pytest, with the two lines that name the backend changed. Everything after this commit changes the fork rather than fwtest, which stays available as the control: the product registry change in particular is not backward compatible, and every backend in this repository already carries its own Framework/, bin/ and DataFormats/. Two headers come over from serial's Framework instead, because fwtest is the minimal backend and does not carry them: ReusableObjectHolder.h and RunningAverage.h, which two of serial's reconstruction modules use. Nothing in pytest uses them either. They are here because a later backend will, and the two Framework/ trees have to stay byte-identical -- a property worth having from the start rather than restoring later by editing this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WP5x6r6bsRbe8kEQEUJk1S
Both were inherited from fwtest and both were silent.
Wrapper<T> brace-initialises its payload, so event.emplace(token, n, 0.0f)
on a std::vector<float> built a *two-element* vector -- {n, 0.0f} read as an
initializer_list -- rather than n elements, and moving one vector into
another did not compile at all. A tag-selected parenthesised constructor
is added for the cases that need it; the brace-initialising one stays,
because an aggregate product needs it.
Event was copyable in declaration only: it owns its products through
unique_ptr, so the implicit copy constructor existed but could never
compile. Nothing copies an Event, so nothing noticed. Deleting it says
what was always true.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WP5x6r6bsRbe8kEQEUJk1S
fwtest keys products on std::type_index alone, so produces<T>() throws if
that type is already registered and one event can hold at most one product
of a type. Comparing two implementations of an algorithm needs two products
of the same type -- one from each -- so the type alone cannot identify a
product.
The alternative is a distinct C++ type per producer, which is what the rest
of pixeltrack-standalone does. That makes the type system carry which
implementation produced a value, so a consumer cannot be written once and
pointed at either, and a third implementation means a third type. Keying on
(type, label) instead is what the demo and CMSSW both do, and it is what
lets a Python module stand in for a C++ one later.
consumes<T>() is now ambiguous when several modules produce T. Rather than
pick one it throws and names the count; a module that wants a particular one
calls consumes<T>(label). The unlabelled form stays because most products
are unique, and keeping it means the ported plugins compile unchanged.
The label is the module's own name. Until the next commits it is the plugin
type, which already separates two different plugins publishing one type.
An empty label means "the only product of that type", so consumes<T>("") and
consumes<T>() are the same lookup. That is what lets a module take the label
of its input from its configuration and pass it on unconditionally: naming an
input becomes something a job does only when it has more than one candidate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WP5x6r6bsRbe8kEQEUJk1S
The schedule and the modules' parameters come from an INI file, parsed with boost property_tree: [options] modules = a, b, c gives the path, and one [label] section per module names its plugin with @type and carries the rest as that module's parameters. test.ini reproduces the workflow fwtest runs from its hard-coded path. A label is now a module's own name rather than its plugin type, so one plugin can be scheduled twice under different labels and its two products tell apart by the (type, label) key. Every module is constructed as Module(ModuleConfig const&, ProductRegistry&). One signature rather than two -- with the configuration optional -- costs an ignored argument in the modules that take no parameters, and buys a compiler error, rather than a silent fallback, when a module that means to read its section declares the wrong constructor. The command line still wins: run-scan.py and the test_* targets drive --numberOfThreads, --numberOfStreams and --maxEvents, so [options] supplies defaults and an explicit switch overrides them. Two mistakes that used to be silent are now errors: an unknown key in [options], and a scheduled label with no section of its own. LIBNAMES selected every directory entry that was not a plugin, which would have turned test.ini into an empty library and linked against it. A library is now a directory that holds C++ sources. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WP5x6r6bsRbe8kEQEUJk1S
A module may now be a Python class. PythonProducer takes `script` and
`factory` from its section and hands the whole section to that factory as a
dict, along with the very ProductRegistry it is being constructed against:
def create(config: dict[str, str], registry) -> object # with .produce(event)
So a Python module declares exactly as a C++ one does, against the same
object, and the tokens it gets back are the only way it can reach a product.
One instance is constructed per stream, each holding its own Python object,
which is what lets the streams run their Python modules concurrently.
The Event offers three ways to publish one, named for what they do:
event.allocate(token) construct it in the Event, nothing set,
and hand back a reference to fill
event.put(token, value) copy in a product that already exists
event.emplace(token, *values) build it from Python values and move it in
allocate() is the zero-copy path: the reference is the product's own storage.
put() and emplace() return nothing, because what is in the Event is not the
object the caller is holding. What emplace() can build is decided per
product from the dictionary -- a scalar, a std::vector of scalars, or an
aggregate whose members are all scalars -- and anything else says so and
names allocate(). A product that can be neither copied nor built says that
too, at run time: the SoA wrappers hold a unique_ptr and are move-only.
Free-threading is not optional: with the GIL enabled the Python modules of
different streams serialise, and a throughput compared against an all-C++ run
would measure that rather than anything about Python. The build refuses to
generate python-config.mk from a GIL-enabled interpreter, and the job prints
the GIL state next to the stream count -- the failure is silent by nature, so
it is made loud in two places. A free-threaded system python3.14t is used
when there is one, otherwise uv installs CPython under external/; either way
what gets embedded is a venv under external/venv.
No binding is written by hand. rootcling reflects the product classes named
in REFLECT_PRODUCTS, and tools/generate_bindings.C walks that reflection
through TClass and writes the nanobind registrations for every product and
everything reachable from one -- public data members, fixed-size arrays as
zero-copy numpy views, public methods -- together with the Event's dispatch
and the name -> type_index table. The product list is written once, in the
Makefile. Nothing under Framework/ names a product type, which is what will
let that directory be identical in every backend.
Three shapes the reflection does not describe on its own, and what the
generator does about them:
- A column that is a pointer and a length. A format keeping its storage as a
unique_ptr<T[]> with the length in another member hands it out as a bare
pointer, and a bare pointer cannot say how far it runs. A public method
returning a std::span -- or an edm::Span, since CMSSW has one -- is a pointer
and a length together, and becomes a numpy view over the column. Getting
std::span past rootcling takes one environment variable: its own option
parser reads -std as -s, and EXTRA_CLING_ARGS is how a flag reaches cling.
- A class reachable only through a method. Following public data members alone
stops at the first wrapper that keeps its storage private, and every method
returning an inner class is then skipped as unbindable, so the generator
follows what public methods hand back as well.
- Two classes wanting the same Python name. The second registration would
quietly shadow the first, so a colliding name keeps its scope.
ROOT also normalises the standard library's namespace away -- a member of type
std::vector<unsigned int> is reported as "vector<unsigned int>", which is not
C++ that compiles. The qualification is put back where the name arrives from
ROOT rather than at each of the places that emit one, so that the two spellings
cannot be collected as two classes and bound twice either.
Methods are bound through lambdas returning decltype(auto). Plain auto deduces
by value, which binds every method returning a reference to a copy: it reads
correctly, and only a write through the reference shows the difference, which
is the kind of bug that surfaces a long way from its cause.
A product that cannot be default-constructed but has a public constructor
wiring it to another product and to a column is built by allocate(token, ...),
from the constructor the reflection reports.
The EventSetup is reached the same way, by type and with no token, through a
dispatch generated from the same product list. The reference it hands out is
not const -- nanobind has no per-instance constness -- and that is worse here
than for the Event, since an EventSetup product is shared by every stream; it
is noted where it is bound. edm::EventSetup turned out to be copyable in
declaration only, exactly as the Event was and for the same reason: it owns its
products through unique_ptr, so the implicit copy constructor existed but could
never compile, and nothing noticed until nanobind tried to instantiate it.
A scalar product needs no dictionary: nanobind's own casters carry it across
by value, exactly as they already carry a class's scalar members, and ROOT
has no TClass for a fundamental type. Those are kept out of the LinkDef,
since rootcling has nothing to generate for them and says so once per rule.
Three things the build gets right rather than silently: the dictionary depends
on every header the product header reaches, taken from the compiler, so an
edit to a class it includes regenerates the bindings; it is linked against the
product libraries, because ROOT loads it on its own and needs the typeinfo and
the vtable of every class it describes -- which is also why the ROOT steps
carry the external library directories themselves, rather than relying on the
environment a job is run in; and a generator that fails stops the build with
its own message instead of leaving the previous output in place to be compiled
against.
ROOT is a build-time dependency only: rootcling and one macro, with nothing
linking against it, so a system installation is used rather than downloading
one. ROOT_HIST=0 keeps the interpreter from writing to $HOME.
Three shared objects, because there are three different things to keep apart.
The generated bindings name every product type, so they are compiled into
edm_core, a Python extension in the library directory that links against the
product libraries and sits above them; the interpreter imports it by name, and
PythonRuntime puts that directory on sys.path first. Nothing under Framework/
names a product -- checked rather than asserted: libFramework has no undefined
symbol mentioning a format. nanobind's amalgamation is compiled exactly once,
into libnanobind.so -- two copies would each keep their own type registry and
casts would stop working across the boundary -- which is what NB_SHARED and
NB_BUILD are for: the first turns its core API from hidden symbols into
exported ones, the second marks the one object that defines them. And
libFramework uses nanobind without owning a module of its own: each shared
object keeps its own pointer to the backend state, filled when *that* object
initialises a module, so Framework's would stay null and the first import from
PythonModuleRunner would segfault. register_module() is the way in -- it joins
the domain edm_core created rather than making a second one.
Everything that touches nanobind is behind PythonModuleRunner's pimpl, so no
plugin translation unit includes it. Nothing requires that -- with the core API
exported, a plugin could use nanobind directly -- but it keeps a future
Python-backed module in any plugin costing nothing extra.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WP5x6r6bsRbe8kEQEUJk1S
Five modules and square.ini: RandomGenerator publishes a vector of random floats, SquareCxx squares it, a PythonProducer running python/square.py squares the same vector with numpy, Compare checks the two element by element and Printer reports. Both squarers publish a std::vector<float> and are told apart only by the label of the module that produced them, which is what the (type, label) key was added for; fwtest could not express this workflow at all. Compare runs with tolerance = 0 and strict = true, so the job fails if the two implementations disagree by a single bit. They do not: x*x is one IEEE multiplication either way, and the Python side is a view of the same C++ storage rather than a copy of it, so there is nothing in between to round. Comparison is a user-defined class rather than a tuple of scalars on purpose: it is what the rootcling/TClass step reflects into bindings, so the generator is exercised by this backend and not only by a real workflow. emplace.ini exercises the other way a Python module publishes: emplace_demo.py builds a std::vector<float> from a list and a pytest::Comparison from four numbers, the C++ Printer reads the aggregate knowing nothing about where it came from, and emplace_check.py reads the vector back. square.py uses allocate() instead, which is the zero-copy path -- it fills the product's own storage through a numpy view -- so between them the two configurations cover both. square.py also reads the EventSetup, by type and with no token: IntESProducer puts a bare int there, and a fundamental type has no TClass, so it crosses through nanobind's own caster exactly as a scalar member of a product does. pytest has had an EventSetup since it forked fwtest and no Python module had ever touched it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WP5x6r6bsRbe8kEQEUJk1S
No binding in this backend is written by hand, which makes the generator the
thing that has to be right: a mistake in tools/generate_bindings.C is a mistake
in every product at once. What there was to check it with was a
std::vector<float> and a four-field aggregate -- a scalar member and a
container, a small corner of what it emits.
Two products carry the shapes a real format has, with none of the content.
pytest::Calibration is conditions, read both one channel at a time -- a method
taking an argument, a class member, a fixed-size array, a static constant --
and as columns. pytest::Samples is a structure of arrays with std::span
columns, a view handed out by reference and a method with a side effect, built
by a constructor that wires it to the conditions and to a column.
bindings.ini runs CalibrateCxx and python/calibrate.py on the same input and
the same conditions, compares the two products column by column with no
tolerance, and reads both back from a second Python module. The arithmetic is
one subtraction and one multiplication, never a multiply-add that
-march=native could contract into an FMA, so numpy and C++ agree bit for bit.
The three interface paths nothing had used are covered here too -- put(),
emplace() of a scalar, and reading a scalar product back -- and so are the two
that have to fail: allocate() of a scalar, put() of something that cannot be
copied.
One of these is a regression test with a story behind it. A method bound as
[](X& o) { return o.method(); } deduces its return type by value, so a method
returning a reference is bound to a copy. Reading through the copy gives the
right answer -- the columns are pointers and still land in the right memory --
so only a write to the view itself shows the difference. samples.view()
followed by view.filled = n is that write, and SamplesCompare fails on the
first event if it goes into a temporary.
The configurations are the tests. test_cpu ran ./pytest --maxEvents 2 with no
configuration at all, which has not been able to do anything since the schedule
started coming from a file; it now runs all four.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WP5x6r6bsRbe8kEQEUJk1S
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend the
fwtestframework with support for a simple configuration language, product labels, and Python interoperability.The Makefile downloads and sets up Python 3.14 with free-threading and nanobind in a
venv, to support efficient multithreading for the Python modules.The
Eventproducts are constructed and owned by the c++Eventclass, while individual modules can be written in C++ or in Python.The
EventSetupproducts are handled in a similar way.Python modules consume and produce wrapped C++ objects with zero copy (except for C++ scalars): this uses nanobind for the runtime bindings, and ROOT/cling dictionaries to generate the custom bindings.
This PR introduces the
pytestapplication: it includes the full framework and some simple C++/Python interoperability tests.