While working on the DSP optimization PRs I wanted to be able to answer, for a
given change: did it actually cost cycles, and did it change the audio? I built an
ESP-IDF benchmark harness for AMY on the ESP32-S3 to do that. Opening this in case
it is useful to you; if not, it is no trouble to keep it outside the repo.
What it does. Compiles a given src/ tree as an ESP-IDF component, renders
fixed deterministic scenes headless, and times each block with both esp_timer and
the CPU cycle counter, alongside a CRC32 of every rendered sample. A driver takes
two git refs, builds both against the same harness, flashes them into two OTA
slots, alternates between them (boot-slot switch plus reset, about a second, so
repeats are cheap), and reports each scene's delta next to its measured noise.
Measured on an S3 at 240 MHz, five captures per side:
| source of error |
measured |
| boot-to-boot, same binary |
+/-0.01% |
| relink, identical sources |
<=0.07% |
So it resolves sub-1% changes, and the CRC indicates whether a change intended to
be output-neutral actually was.
Relationship to the loadsweep hwci
These measure different things, and I do not think this replaces the loadsweep:
- loadsweep measures the firmware you ship: the real audio task, both cores,
I2S DMA contending for memory bandwidth, real interrupt load. It answers "how
many voices before we overrun?" - a capacity question, and it can only be
answered with the whole system running.
- this harness measures a single render call in isolation: audio off, one core,
no DMA. It answers "did this change cost cycles?" - a differential question,
where the isolation is what buys the precision.
Limitations worth stating up front: the isolation that produces the +/-0.01% is
also an unquantified bias. It is single-core where production renders on two, and
it has no I2S DMA competing for PSRAM bandwidth. A change that trades memory
traffic for arithmetic could therefore rank differently here than in the real
firmware. The relative deltas are trustworthy; the absolute numbers, and possibly
the ranking of memory-bound changes, may not transfer to production. That is why I
would not propose it as a substitute for measuring the shipped path.
One structural difference: because the timing lives outside src/, the harness can
point at an arbitrary git ref and measure it unmodified. The loadsweep
instrumentation lives in src/i2s.c behind #ifdef ARDUINO_SPEEDTEST, so it can
only measure trees that carry that block.
It needs two small #ifndef guards in src/amy.h to build at a chosen sample rate
and in float/fixed. It injects these into the scratch copies of the test refs, since passing
these as D flags-DAMY_SAMPLE_RATE would be overwritten unconditionally.
You are free to use, retool or cannibalize this in any way you see fit - or not at all.
it builds both sides headlessly and emits JSON, and the flash-and-capture half could fit the same
self-hosted runner pattern the loadsweep hwci uses.
If it´s too niche for hwci - it could still be convenient for manual use during performance work.
While working on the DSP optimization PRs I wanted to be able to answer, for a
given change: did it actually cost cycles, and did it change the audio? I built an
ESP-IDF benchmark harness for AMY on the ESP32-S3 to do that. Opening this in case
it is useful to you; if not, it is no trouble to keep it outside the repo.
What it does. Compiles a given
src/tree as an ESP-IDF component, rendersfixed deterministic scenes headless, and times each block with both
esp_timerandthe CPU cycle counter, alongside a CRC32 of every rendered sample. A driver takes
two git refs, builds both against the same harness, flashes them into two OTA
slots, alternates between them (boot-slot switch plus reset, about a second, so
repeats are cheap), and reports each scene's delta next to its measured noise.
Measured on an S3 at 240 MHz, five captures per side:
So it resolves sub-1% changes, and the CRC indicates whether a change intended to
be output-neutral actually was.
Relationship to the loadsweep hwci
These measure different things, and I do not think this replaces the loadsweep:
I2S DMA contending for memory bandwidth, real interrupt load. It answers "how
many voices before we overrun?" - a capacity question, and it can only be
answered with the whole system running.
no DMA. It answers "did this change cost cycles?" - a differential question,
where the isolation is what buys the precision.
Limitations worth stating up front: the isolation that produces the +/-0.01% is
also an unquantified bias. It is single-core where production renders on two, and
it has no I2S DMA competing for PSRAM bandwidth. A change that trades memory
traffic for arithmetic could therefore rank differently here than in the real
firmware. The relative deltas are trustworthy; the absolute numbers, and possibly
the ranking of memory-bound changes, may not transfer to production. That is why I
would not propose it as a substitute for measuring the shipped path.
One structural difference: because the timing lives outside
src/, the harness canpoint at an arbitrary git ref and measure it unmodified. The loadsweep
instrumentation lives in
src/i2s.cbehind#ifdef ARDUINO_SPEEDTEST, so it canonly measure trees that carry that block.
It lives here
It needs two small
#ifndefguards insrc/amy.hto build at a chosen sample rateand in float/fixed. It injects these into the scratch copies of the test refs, since passing
these as D flags
-DAMY_SAMPLE_RATEwould be overwritten unconditionally.You are free to use, retool or cannibalize this in any way you see fit - or not at all.
it builds both sides headlessly and emits JSON, and the flash-and-capture half could fit the same
self-hosted runner pattern the loadsweep hwci uses.
If it´s too niche for hwci - it could still be convenient for manual use during performance work.