Eval-gated quality harness for AI-generated eye-care SOAP notes, with line-level provenance.
AI-generated clinical notes need a QA spine: ScribeGate uses deterministic evals as regression tests to catch hallucinations, missing details, and terminology errors before a note leaves the system. Every line in a generated note is traceable to its source transcript—click a note line in the UI and see the exact character span it came from. This provenance chain is the foundation for human review.
Transcript (text file)
|
v
Generator (mock-only by default; Claude API env-gated via SCRIBEGATE_USE_API=1)
|---> Generates SOAP note with char-span references to source
v
Normalizer
|---> Checks VA format, IOP units, laterality, Rx syntax
v
Judge (mock scoring engine; Haiku API available env-gated)
|---> Scores completeness, hallucination, coding_plausibility, terminology (1-5 scale)
|---> Computes aggregate (0-1 normalized)
v
Router (confidence thresholds)
|---> aggregate >= 0.85 + no error violations → auto_accept
|---> 0.60 <= aggregate < 0.85 + no error violations → review
|---> otherwise → regenerate
v
Drift detection + eval gate | Analytics/ROI | Correction loop (reviewer edits → golden examples) | Audit dossier + integrity hashes | Consent-gated live encounter capture (reference-free judging)
v
Provenance view + append-only decision log
Mock mode is deterministic by default. Claude API backends are never required and only activate when SCRIBEGATE_USE_API=1 is set; default path works offline.
- Start here: Default landing page — a five-minute guided tour linking straight to each other page, plus a full glossary.
- Overview: Dashboard landing page showing synthetic-data banner and key metrics.
- Analytics: Per-visit-type benchmark summary, mean scores, and routing breakdown (auto_accept / review / regenerate).
- Drift: Rolling-window quality regression detection and alert status.
- Data moat: Moat curve (aggregate + auto-accept rate per golden generation), headline golden-set/correction/benchmark-lift metrics, and a "Promote pending candidates" button that closes the correction -> promotion -> new generation -> re-benchmark loop live.
- Judge calibration: Mean CI95 width per visit type from the probabilistic-judge instrumentation, CI-aware vs. point-estimate routing deltas, and a per-case table.
- Review queue: Generated notes routed to
review(0.60–0.85 aggregate); click to view provenance and make line-level corrections. - Provenance: Click any SOAP line to highlight the exact transcript character spans it traces to (one or many spans stitched into a single line).
- Live encounter: Browser mic input → speech-to-text → eval pipeline; microphone disabled until provider and patient consent is both recorded to the audit log.
- Live mode: Budget-capped, passcode-gated page that calls the real Claude API end-to-end (draft → judge → route) instead of the offline mock, with a per-note cost breakdown.
- Economics: Unit-economics engine — cheapest model tier clearing the quality floor, its margin vs. the premium tier, and a live-recomputing haiku/sonnet/opus comparison table.
- About: Product story, synthetic-data notice, link to PRODUCTION_PATH.md.
Cold-start in under 5 minutes:
git clone <repo-url>
cd scribegate
make install # creates .venv (python3.12) and installs requirements
# Generate notes, run judges, route decisions
.venv/bin/python -m scribegate.cli run --all
# Aggregate per-visit-type benchmark report
.venv/bin/python -m scribegate.benchmark
# Launch interactive dashboard (12-page demo app — see Feature Tour)
.venv/bin/streamlit run app/streamlit_app.py
# optional: mic capture with browser speech engine (streamlit-mic-recorder in requirements)
# demo works fully via text entry without a mic; Chrome recommended if using mic
# Run full test suite (395 tests)
make testThere is no hosted CI; all checks run on your machine.
make install # one-time: .venv + dependencies (override interpreter: make install PYTHON=python3.11)
make check # pipeline -> pytest -> benchmark -> baseline-floor drift gate (what CI used to do)
make hooks # one-time: git config core.hooksPath .githooks (pre-push runs `make check`)
git push --no-verify # skip the pre-push check when you mustIndividual targets: make pipeline, make test, make eval. make check regenerates ignored files under data/results/ and rewrites the tracked benchmark.md and econ_matrix.json; discard those with git checkout data/results/ unless you intend to commit them.
Current results on 20 synthetic transcripts (4 visit types × 5 cases each), generated and judged by mock backends:
| Visit Type | N | Completeness | Hallucination | Coding Plausibility | Terminology | Mean Aggregate | auto_accept | review | regenerate |
|---|---|---|---|---|---|---|---|---|---|
| comprehensive_exam | 5 | 3.00 | 5.00 | 4.80 | 5.00 | 0.86 | 4 | 1 | 0 |
| glaucoma_followup | 5 | 3.20 | 5.00 | 4.20 | 5.00 | 0.84 | 2 | 3 | 0 |
| cataract_postop | 5 | 2.40 | 5.00 | 4.80 | 4.40 | 0.79 | 2 | 1 | 2 |
| contact_lens_fitting | 5 | 3.20 | 3.20 | 5.00 | 5.00 | 0.78 | 1 | 4 | 0 |
Reading the table: Completeness tops out at 3 because the mock generator is deliberately extractive, not synthesizing—it pulls content directly from the transcript. Contact-lens scores lowest by design; the fixture set is intentionally messy (colloquial dictation, crosstalk, loose structure) to stress-test the harness. Low scores on noisy input mean the eval gate is working correctly, not that the pipeline is broken.
make eval (part of make check) runs the full pipeline (python -m scribegate.cli run --all) and fails if quality falls below committed baseline floors in specs/baseline.json. The drift detector tracks the rolling-window history in data/results/history.jsonl for operational insight; the hard gate compares only the latest run against static baselines so a single regression cannot slip through. A degraded-model demo row in history_demo.jsonl shows the gate catching a silent model downgrade.
scribegate/
__init__.py # Package marker
cli.py # Entry point: generate → normalize → judge → route
generator.py # Mock drafter + critics (Claude API optional, env-gated)
normalizer.py # VA format, IOP, laterality, Rx rules per specs/terminology.yaml
judge.py # Mock scorer or Haiku API (line alignment, similarity scoring)
router.py # Route decision logic: thresholds from specs/rubric.yaml
benchmark.py # Aggregate results into per-visit-type markdown table
drift.py # Rolling-window regression detection + CI baseline gate
analytics.py # Per-visit-type ROI and failure-mode analytics
corrections.py # Reviewer line-level corrections → candidate golden examples
audit.py # Dossier assembly: note + scores + corrections + hashes
moat.py # Golden-set generations: promotion, re-benchmark, moat curve
calibration.py # Probabilistic-judge sampling, CI95, CI-aware routing
live.py # Live-mode orchestrator: real, budget-guarded Claude API calls
costs.py # Pricing table + append-only cost ledger for live.py
economics.py # Unit-economics engine: cost/margin per model tier
specs/
INTERFACES.md # Data shapes and module contracts (frozen)
terminology.yaml # Validation rules for VA, IOP, laterality, Rx
rubric.yaml # Judge scoring anchors and router thresholds
baseline.json # CI eval gate floors per metric
consent_copy.yaml # UI strings for live-capture consent gate
pricing.yaml # Per-model-tier USD/MTok pricing (editable source of truth)
ui_copy.yaml # Plain-language page titles/copy for the Streamlit UI
data/
transcripts/ # Synthetic clinic visit recordings (*.txt)
golden_notes/ # Reference SOAP notes (*.json) for comparison
results/
*.json # Generated note + judge result + route per transcript
decision_log.jsonl # Append-only log; CLI route events + reviewer decisions
candidate_golden.jsonl # Reviewer corrections (immutable append-only)
history.jsonl # Rolling-window history for drift detection
benchmark.md # Aggregated per-visit-type results
app/
streamlit_app.py # Dashboard: Benchmark, Drift, Review queue, Provenance, Live capture
views/ # Streamlit page modules
common.py # Shared UI/data helpers
.streamlit/ # Streamlit config (secrets, theme)
tests/
test_*.py # 353 unit & integration tests (mock-only, no API calls)
- Not a scribe product. ScribeGate is a QA harness for evals and provenance, not a note-generation system ready for clinical use.
- Not clinical advice. Synthetic data only; no real patient records, no EHR integration, no regulatory clearance.
- Not production-ready. This is a reference implementation demonstrating eval-gated quality gates. Hardening for production (HIPAA, audit trails, live API backends, monitoring) is documented separately.
streamlit run app/streamlit_app.py) renders its own "100% SYNTHETIC data — no PHI" banner so the warning travels with the UI, not just this file.
For the production story (hardening roadmap, provenance-in-production, EHR adapter pattern), see PRODUCTION_PATH.md in this repo.