TLAPS Proof Benchmark

A benchmark for evaluating AI's ability to write TLAPS (TLA+ Proof System) proofs.

Benchmark results are available on the TLAPS-Bench website.

Overview

TLAPS proofs are checked mechanically by tlapm: a proof is either accepted or rejected, with no partial credit and no room for a plausible-but-wrong argument. That makes proof construction a sharp test of an AI's formal reasoning.

Each task presents a TLA+ theorem whose proof body is replaced by PROOF OBVIOUS; the AI must replace it with a real proof that tlapm accepts. Tasks come in two types:

• Proof completion (--mode proof-completion) — the model and proof scaffold (inductive invariants, lemma decomposition, and preceding lemmas marked PROOF OMITTED) are given as read-only context, and the AI fills in one marked target proof.
• Proof from scratch (--mode proof-from-scratch) — only the model and the target theorem statement remain; the AI must invent the entire proof structure, including any helper lemmas.

Benchmark problems

The benchmark draws on two kinds of source. A base of classic TLA+ example libraries, a small set of systems specifications — real protocols, several with no published proof. Both sets are expected to keep growing as more specifications are added. A – marks a source with no human proofs, from which no proof-completion task can be derived.

Example libraries

Source	Examples	Proof completion	Proof from scratch	Total
tlaplus/Examples	44	357	149	506
TLAPS distribution examples	14	91	51	142
Subtotal	58	448	200	648

Systems specifications

Source	Examples	Proof completion	Proof from scratch	Total
apalache-examples (Konnov)	2	257	–	257
ZooKeeper (Remix)	2	–	18	18
Ivy liveness	6	–	12	12
etcd (Specula)	1	–	8	8
OpenAddressing	1	1	5	6
two_thread_mutex (Anvil)	1	–	1	1
Subtotal	13	258	44	302

71 examples, 950 tasks in total. A per-example breakdown is in docs/DATASET.md.

Running

Requirements: uv and Docker. Windows users should run the benchmark through WSL2; native Windows is not supported.

Recommended hardware

Proof checking can use substantial memory, especially for Isabelle-heavy tasks. A few of these tasks can use significantly more than 64 GB of RAM, even when only one job is running.

Use the following as a rough capacity-planning reference per parallel job:

Profile	vCPUs per job	RAM per job	Guidance
Recommended	8–12	96 GB	Provides better memory headroom.
Lower-headroom	8–12	64 GB	A starting point; some Isabelle-heavy tasks may require more.

On a small machine, start with --jobs 1. Increase the value after you monitor peak memory use.

Run the benchmark

git clone https://github.com/specula-org/tlaps-bench.git
cd tlaps-bench
export OPENAI_API_KEY=sk-...        # This step is optional: Codex is the default backend if no OpenAI key is provided.
uv run tlaps-bench run --filter GCD_GCD3

The first run builds a sandbox Docker image (tlapm, SANY, and the proof checker bundled in) and runs the task inside it — a firewall allows only the LLM API hosts, and the benchmarks are mounted read-only. Later runs reuse the image. Results land in results/<mode>/<backend>/<timestamp>/.

Scale up, or switch task type:

# Proof-completion Core: 190 selected tasks, 4 in parallel
uv run tlaps-bench run --task-list core --jobs 4 --timeout 7200

# Full proof-completion suite: 4 in parallel, 2h timeout each
uv run tlaps-bench run --jobs 4 --timeout 7200

# Proof from scratch
uv run tlaps-bench run --mode proof-from-scratch --jobs 4

Each run writes results.json and summary.md (with specification pass rate as the primary score and task-level pass rate as a secondary metric); uv run tlaps-bench score (re)computes and compares scores. See the scoring documentation for the grouping rule and formulas. Use --resume with a fixed --output-dir to skip tasks already recorded as PASS, and --force-build to rebuild the image after changing source.

Choosing an agent (--backend / --model) and its credentials, the full CLI reference, and native (--no-container) setup are covered in the usage guide.

License

MIT — see LICENSE. Third-party benchmark sources are attributed in NOTICE.