README.md

Benchmarks

This folder contains pytest-benchmark-based micro-benchmarks for the public Python API.

Before running any benchmark, ensure you have built the project with the dedicated profile:

Pytest

Useful commands

Run all benchmarks at once with the following command.

In all cases, gc always disabled, warmup always enabled:

uv run pytest benchmarks/ `
--benchmark-only `
--benchmark-warmup=true `
--benchmark-disable-gc

Group output by parametrized dataset size:

uv run pytest benchmarks/ `
--benchmark-only `
--benchmark-group-by=param:size `
--benchmark-warmup=true `
--benchmark-disable-gc

Save a run with a readable name:

uv run pytest benchmarks/ `
--benchmark-only `
--benchmark-warmup=true `
--benchmark-disable-gc `
--benchmark-save=iter-sizes

Save stats plus raw timing data:

uv run pytest benchmarks/ `
--benchmark-only `
--benchmark-save-data `
--benchmark-autosave

Compare against the latest saved run:

uv run pytest benchmarks/ `
--benchmark-only `
--benchmark-compare

Advanced examples

uv run pytest benchmarks/test_sequences.py::test_init `
--benchmark-only `
--benchmark-group-by=param:size `
--benchmark-group-by=name `
--benchmark-warmup=true `
--benchmark-disable-gc `
--benchmark-columns=median,mean,min,max,stddev `
--benchmark-sort=mean `
--benchmark-compare

Run a single test, grouped by size + name, compare against the last saved run, with warmup and GC disabled, and show only the median, mean, min, max, and stddev columns.

Also:

uv run pytest benchmarks/test_sequences.py::test_init `
-k "10-" `
--benchmark-only `
--benchmark-group-by=name `
--benchmark-warmup=true `
--benchmark-disable-gc `
--benchmark-columns=median,mean,min,max,stddev `
--benchmark-sort=mean `
--benchmark-compare

Here we add a k filter to only run the 10- element size, and group by name only.

Comparing multiple runs:

uv run pytest-benchmark compare `
  ".benchmarks\Windows-CPython-3.13-64bit\0055_map_juxt_dev.json" `
  ".benchmarks\Windows-CPython-3.13-64bit\0056_map_juxt_tup.json" `
  ".benchmarks\Windows-CPython-3.13-64bit\0057_map_juxt_tup_small_vec.json" `
  --between=median `
  --group-by=param `
  --sort=name `
  --time-unit=us

Saved format

With specific name:

.benchmarks/<platform-python>/<prefix_id>_<name>.json

With autosave:

.benchmarks/<platform-python>/<prefix_id>_<commit>_<timestamp>.json

• --benchmark-save and --benchmark-autosave save JSON benchmark reports
• --benchmark-save-data includes raw timing samples in those JSON files
• --benchmark-json path.json writes a full JSON report to a path you choose
• --benchmark-storage=<path> changes the default storage path for saved runs

sortedcontainers vs pyochain

For these benchmarks in particular, run uv run -m benchmarks, and add --help for more informations.

---

Import benchmark

If you want to check import speed, you can use the builtin python command:

uv run python -X importtime -c "import pyochain"

To get a table with sorted import times:

uv run python -X importtime -c "import pyochain" 2>&1 |
Select-String "import time:" |
Where-Object { $_ -notmatch "cumulative" } |
ForEach-Object {
    $parts = $_ -split '\|'
    [PSCustomObject]@{
        Self = [int](($parts[0] -replace '.*:').Trim())
        Cumulative = [int]($parts[1].Trim())
        Module = $parts[2].Trim()
    }
} |
Sort-Object Cumulative -Descending |
Format-Table -AutoSize

Sources

• Docs overview: https://pytest-benchmark.readthedocs.io/en/latest/
• Usage and CLI options: https://pytest-benchmark.readthedocs.io/en/latest/usage.html
• Comparing saved runs: https://pytest-benchmark.readthedocs.io/en/latest/comparing.html
• FAQ on noisy results: https://pytest-benchmark.readthedocs.io/en/latest/faq.html
• Upstream source: https://github.com/ionelmc/pytest-benchmark