PRML v0.1 — Pre-Registered ML Manifest Specification — Request for Public Review

[sk8ordie84]

PRML v0.1 — Pre-Registered ML Manifest Specification — Request for Public Review

The first working draft of PRML (Pre-Registered ML Manifest) is published for public review.

• Spec: https://spec.falsify.dev/v0.1
• Source: spec/PRML-v0.1.md
• Reference implementation: falsify (this repo, MIT)
• License: CC BY 4.0
• Editor: Studio-11 — hello@studio-11.co

---

What it is

A content-addressed serialization format for pre-registered ML evaluation claims. A PRML manifest binds — to a single SHA-256 digest, computed before the experiment runs — five things:

1. the metric (accuracy, f1, auroc, …)
2. the comparator + numeric threshold (>= 0.85)
3. the dataset content hash
4. the random seed
5. the producer identity

A verifier with the manifest, the dataset, and the model can independently recompute the digest, execute the claim, and emit a deterministic verdict (PASS / FAIL / TAMPERED). Honest amendments are recorded in a forward-only chain via prior_hash.

Why now

EU AI Act Article 12 (logging) and Article 18 (10-year retention of technical documentation) enter force August 2, 2026. NIST AI RMF and ISO/IEC 42001:2023 reference content-addressed audit trails as a recommended control. The intersection — cryptographic, pre-experimental, ML-evaluation-shaped — is empty. PRML is one attempt to fill it.

What I'm asking for

This is a draft, not a finished standard. Three concrete asks:

1. Format review — is the canonical serialization unambiguous? Are there YAML 1.2 edge cases the spec misses?
2. Threat model gaps — six adversaries are enumerated in §6 of the paper outline. What's missing?
3. Compliance mapping correctness — §9 of the spec maps fields to AI Act / NIST / ISO obligations. Practitioners and lawyers: are the bindings defensible?

What this is not

• A product pitch. The spec is independent of any tool, language, or platform.
• A finished standard. v0.2 is planned for 2026 Q4 with algorithm agility, signature mandate for high-risk systems, and a formal canonicalization grammar.
• A replacement for model cards, datasheets, or red-team reports. PRML underpins them.

How to engage

• Comments on this thread — high-level direction, scope.
• GitHub Issues with the spec label — concrete bugs, ambiguities, missing fields.
• Pull requests against spec/PRML-v0.1.md — proposed edits with rationale.
• Email hello@studio-11.co — anything you'd rather not say in public.

A v0.2 freeze is targeted for 2026-05-22, with a test vector suite and BNF grammar for the canonicalizer. Substantive review by then has the highest leverage.

Thanks for reading.

[sk8ordie84]

Weekend update — second implementation, portability findings, v0.2 roadmap, preprint

Three weeks before the v0.2 freeze, the public artifacts have grown. Posting here for visibility:

• Second reference implementation: impl/js/falsify.js — 404 LOC Node.js, hand-rolled canonicalizer, 12/12 v0.1 vectors pass byte-for-byte. Zero runtime deps beyond Node.js stdlib.
• Portability findings: spec/analysis/canonicalization-portability-v0.1.md — three cross-language gotchas surfaced by the JS implementation (uint64 precision, integer-valued float typing, plain-scalar quoting heuristics). Each one motivates a specific v0.2 change.
• v0.2 RFC roadmap: spec/v0.2/ROADMAP.md — ten changes, five open RFC questions, freeze targeted 2026-05-22.
• Positioning matrix: spec/analysis/positioning-v0.1.md — PRML vs in-toto / SLSA / Model Cards / HELM / ClinicalTrials.gov.
• Compliance landing: spec/compliance/landing.md — for compliance leads, AI governance officers, notified body assessors.
• arXiv preprint: spec/paper/ — 14-page LaTeX, CC BY 4.0. Submission to cs.CR is in flight.

What I'm asking for:

1. RFC-Q-01 through Q-05 in the v0.2 roadmap — explicit fork-in-the-road decisions that benefit from outside opinion.
2. Portability findings review — if you've implemented PRML in any language other than Python, divergences welcome.
3. Compliance review — the field-to-Article mapping in §7.1 of the preprint deserves an eye from anyone who has audited an ML accuracy claim retroactively.

v0.2 freeze in 21 days. Substantive review carries the most weight in that window.

[sk8ordie84]

Update: third reference implementation in Go, all 12 vectors pass byte-for-byte

Following the JavaScript second implementation posted earlier today, I built a third reference implementation in Go this evening: impl/go/falsify.go — ~450 LOC, Go standard library only, no external dependencies. All 12 v0.1 conformance vectors pass byte-for-byte on first compile.

The interesting empirical finding: Go's stdlib encoding/json with Decoder.UseNumber() handles two of the three portability gotchas natively, with no workaround. Specifically:

• TV-006 (max uint64 seed): json.Number preserves the raw decimal string, no precision loss.
• TV-008 (threshold: 1.0 round-trip): json.Number preserves the raw .0 suffix, no integer-vs-float type ambiguity.
• TV-008 (== plain-scalar): same hand-rolled predicate as JS — Go YAML libraries also diverge from PyYAML here, so the predicate had to be re-implemented manually.

So the severity of the v0.1 portability gotchas is language-stdlib-dependent. The portability findings doc has been updated with a per-language severity table: spec/analysis/canonicalization-portability-v0.1.md.

With three independent implementations across Python (PyYAML), JavaScript (hand-rolled stdlib), and Go (hand-rolled stdlib) all reproducing the canonical bytes byte-for-byte, the v0.1 spec is no longer "what PyYAML happens to emit" — it is "what the conformance suite says, reproducibly across languages."

Looking for: a fourth implementation in Rust (or Java, or anything else). The 12 vectors are the contract. PRs against impl/<language>/ welcome.

[sk8ordie84]

v0.1.2 release + Rust 4th implementation + Finding 4 patches + dev.to write-up

Quick announcement on the day's releases.

Released today: v0.1.2 — published on PyPI (pip install falsify==0.1.2) and tagged at https://github.com/sk8ordie84/falsify/releases/tag/v0.1.2. Code unchanged from 0.1.1; the milestone is the specification work.

What landed since the morning update:

• Fourth reference implementation in Rust — impl/rust/, ~600 LOC, two crate dependencies. Passes 12/12 v0.1 + 6/6 v0.2 candidate vectors byte-for-byte.
• Finding 4 patched in JavaScript and Go — small-magnitude floats (TV-018) now reproduce PyYAML's 1.0e-06 form across all four implementations. The interim patch is documented; v0.2's always-quoted-numbers (RFC-Q-04) remains the strategic fix.
• Multi-language conformance CI — new workflow runs all four implementations against both vector suites on every push, PR, and daily at 04:00 UTC. Total verification surface: 4 × 18 = 72 byte-for-byte agreements asserted per run.
• Blog post on dev.to documenting the three v0.1 portability findings end-to-end: https://dev.to/sk8ordie84/i-implemented-prml-in-two-languages-three-things-broke-that-the-spec-didnt-warn-about-2j62

Total state today:

• 4 reference implementations (Python · JavaScript · Go · Rust)
• 12 v0.1 normative vectors + 6 v0.2 candidates, all 4 langs pass byte-for-byte
• 4 documented portability findings, all patched
• 564 unit tests across the test suite, all green

What I'm still asking for, unchanged: review of the v0.2 RFC roadmap, particularly the five open RFC questions before the 2026-05-22 freeze. Substantive critique on the field-to-Article mapping for AI Act compliance also welcome.