Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
f6c84dc
chore: 🔧 ignore build output, env files, and agent scratch
AndrewDongminYoo Aug 18, 2026
34e3081
build: 🏗️ set up the pnpm workspace and the contract package
AndrewDongminYoo Aug 18, 2026
77de059
fix: 🔧 add types and allowImportingTsExtensions to tsconfig for type …
AndrewDongminYoo Aug 18, 2026
bf1c177
feat(contract): ✨ split OCR text into indexed evidence lines
AndrewDongminYoo Aug 18, 2026
2b16c0b
docs(plans): 📝 correct Task 2's line-index semantics to match the Dar…
AndrewDongminYoo Aug 19, 2026
667f2fd
fix(contract): evidence lines index counts recognised lines only
AndrewDongminYoo Aug 19, 2026
e1b332b
fix(contract): correct OcrEvidence JSDoc to match implementation
AndrewDongminYoo Aug 19, 2026
2398cb9
feat(contract): ✨ parse receipt dates and reject expiries
AndrewDongminYoo Aug 19, 2026
7f7cf5c
docs(plans): 📝 make Task 3 read every date match, as the Dart source …
AndrewDongminYoo Aug 19, 2026
dd95224
fix(contract): 🐛 parseDate skips calendar-invalid matches like Dart's…
AndrewDongminYoo Aug 19, 2026
19ef22f
feat(contract): ✨ read receipt amounts as integer minor units
AndrewDongminYoo Aug 19, 2026
5acfdcf
docs(plans): 📝 cite the Dart range where amount parsing actually lives
AndrewDongminYoo Aug 19, 2026
2cdba4d
docs(contract): 📝 document DATE_PATTERN_G reuse hazard and minorUnits…
AndrewDongminYoo Aug 19, 2026
e59b856
docs(plans): 📝 stop asserting a fare row becomes the selected total
AndrewDongminYoo Aug 19, 2026
5fb53b8
feat(contract): ✨ pick the paid total apart from discounts and counts
AndrewDongminYoo Aug 19, 2026
18304b8
feat(contract): ✨ infer the receipt currency from its markers
AndrewDongminYoo Aug 19, 2026
2262d67
fix(contract): 🔧 drop dead import, tighten typecheck to catch unused …
AndrewDongminYoo Aug 19, 2026
617bfd0
test(contract): 🐛 pin the _isPricedItem gap with a test that can fail
AndrewDongminYoo Aug 19, 2026
8f6182b
feat(contract): ✨ extract priced line items with their evidence
AndrewDongminYoo Aug 19, 2026
df3b958
fix(contract): 🔧 close the isPricedItem gap in cents-row currency inf…
AndrewDongminYoo Aug 19, 2026
e5ee163
fix(contract): 🐛 correct items.ts source citations
AndrewDongminYoo Aug 19, 2026
a86aca5
feat(contract): ✨ assemble the deterministic receipt parser
AndrewDongminYoo Aug 19, 2026
53e93af
fix(contract): 🎯 skip amount/date lines when selecting the merchant
AndrewDongminYoo Aug 19, 2026
09f97b6
docs(contract): 📝 correct REFERENCE_LABEL duplication comment in anal…
AndrewDongminYoo Aug 19, 2026
f29c18b
test(contract): ✅ gate the parser on the 12-receipt corpus
AndrewDongminYoo Aug 19, 2026
a2e2e88
docs: 📝 replace the claimed parser baseline with the measured one
AndrewDongminYoo Aug 19, 2026
bdec458
fix(contract): 🎯 pin items divergence, fix KR-04 merchant sign, close…
AndrewDongminYoo Aug 19, 2026
570379f
test(contract): 🔒 pin exact values for non-derivable merchant/paidTot…
AndrewDongminYoo Aug 19, 2026
4fd2142
feat(contract): ✨ verify model values against their evidence
AndrewDongminYoo Aug 19, 2026
fe574e5
docs(plans): 📝 make the guard's flagship test exercise both halves
AndrewDongminYoo Aug 19, 2026
40746ca
test(contract): pin the excerpt-real-but-value-wrong hallucination case
AndrewDongminYoo Aug 19, 2026
f0c6dc8
fix(contract): match evidence per line, not across the whole page
AndrewDongminYoo Aug 19, 2026
f6ffe13
fix(contract): verifyEvidence rejects an empty or whitespace-only exc…
AndrewDongminYoo Aug 19, 2026
517d24b
feat(contract): ✨ recompute the item sum against the claimed total
AndrewDongminYoo Aug 19, 2026
ecd8218
feat(contract): ✨ anchor an evidence excerpt to its OCR box
AndrewDongminYoo Aug 19, 2026
6cfc70a
refactor(contract): extract shared normalize function
AndrewDongminYoo Aug 19, 2026
81cbd86
feat(contract): ✨ define the model reply schema once, in zod
AndrewDongminYoo Aug 19, 2026
89e7da7
docs(plans): 📝 drop the currency branch and make the strictness test …
AndrewDongminYoo Aug 19, 2026
0b257ba
fix(contract): drop dead currency field, make quantity optional, pin …
AndrewDongminYoo Aug 19, 2026
75222a4
feat(web): ✨ extract a receipt through the parser, the model, and the…
AndrewDongminYoo Aug 19, 2026
b83c0a2
fix(web): report a rejected model reply instead of vanishing it into …
AndrewDongminYoo Aug 19, 2026
258d54b
test(web): assert toStrictSchema against the real emitted schema, rec…
AndrewDongminYoo Aug 19, 2026
11660cf
fix(web): read purchaseDate back through local getters, not toISOString
AndrewDongminYoo Aug 19, 2026
23d64c0
feat(web): ✨ show extraction results with their evidence
AndrewDongminYoo Aug 19, 2026
8b79fdd
docs(plans): 📝 add the image fallback as a task, since nothing implem…
AndrewDongminYoo Aug 19, 2026
378dc3e
docs: 📝 add CLAUDE.md for future sessions in this repository
AndrewDongminYoo Aug 19, 2026
36dc41b
feat(web): ✨ send a page's image when the caller attached one
AndrewDongminYoo Aug 22, 2026
4a5ac00
refactor(contract): 🚚 move the response types where both clients can …
AndrewDongminYoo Aug 22, 2026
a39ed70
feat(mobile): ✨ scan a receipt and show its evidence on the image
AndrewDongminYoo Aug 22, 2026
09cea88
docs: 📝 lead the README with the measured parser baseline
AndrewDongminYoo Aug 22, 2026
1778835
fix(web): 🐛 answer a malformed request with 400, not an unhandled 500
AndrewDongminYoo Aug 22, 2026
a5d8c9a
docs: 📝 stop restating the test count in the README
AndrewDongminYoo Aug 22, 2026
76b4d3f
fix(contract): 🐛 check what a value claims, not just where it was quo…
AndrewDongminYoo Aug 22, 2026
fe4ba18
fix(web): 🐛 stop uploading the receipt photo without being asked
AndrewDongminYoo Aug 22, 2026
06660d6
docs: 📝 correct every document that named the removed guard
AndrewDongminYoo Aug 22, 2026
5daa09d
build(mobile): ⬆️ adopt the versions the first native build actually …
AndrewDongminYoo Aug 23, 2026
03a918a
fix(mobile): 🐛 derive the API host from the dev server, not from loca…
AndrewDongminYoo Aug 23, 2026
5a13d17
fix(contract): 🐛 verify every part of a claim, and stop a gate that c…
AndrewDongminYoo Aug 23, 2026
ccdd896
fix(web): 🐛 read the whole scan as one document, and stop the server'…
AndrewDongminYoo Aug 23, 2026
006c635
fix(web): 🐛 close the consent, image and error-label holes at the bou…
AndrewDongminYoo Aug 23, 2026
69f4304
fix(contract): 🐛 refuse an amount JS cannot hold, and a box that coul…
AndrewDongminYoo Aug 23, 2026
11588b0
docs: 📝 close the drift the review found between docs, config and code
AndrewDongminYoo Aug 23, 2026
c1183ac
fix(web): 🐛 let only the newest file selection write the image
AndrewDongminYoo Aug 23, 2026
ea9d221
fix(web): 🐛 judge the request before the server's own configuration
AndrewDongminYoo Aug 23, 2026
4714afc
fix(mobile): 🐛 take the API host from the bundle URL, not from a mani…
AndrewDongminYoo Aug 23, 2026
7a26d97
fix(contract): 🐛 read a Korean label whose printer spaced its characters
AndrewDongminYoo Aug 23, 2026
5b06eb1
feat(contract): ✨ let evidence span the adjacent lines a receipt prin…
AndrewDongminYoo Aug 23, 2026
f1b4b3a
fix(contract): ✨ raise the evidence cap to the four lines a real rece…
AndrewDongminYoo Aug 24, 2026
0a084d4
fix(contract): 🐛 stop a receipt's VAT shipping as its paid total
AndrewDongminYoo Aug 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: ci

on: [push, pull_request]

# The job only reads the repository; nothing here writes to it.
permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Comment thread
coderabbitai[bot] marked this conversation as resolved.
with:
# checkout leaves its token in .git/config by default, where any
# later step (or a dependency's install script) could reuse it.
persist-credentials: false
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 24
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm test
- run: pnpm typecheck
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
node_modules/
.next/
next-env.d.ts
.expo/
.superpowers/
*.tsbuildinfo
.DS_Store
.env*
!.env.example
67 changes: 67 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Read `docs/specs/2026-08-19-receipt-evidence-design.md` for why this exists and what the pipeline is, `docs/notes/corpus-baseline.md` for the measured numbers the README quotes, and `docs/plans/2026-08-19-receipt-evidence.md` for the task-by-task build.
This file covers what those do not, or what will bite before you get to them.

## Commands

```bash
pnpm test # node --test over packages/*/test and apps/*/test
pnpm typecheck # tsc over the workspace, then apps/mobile's own tsconfig
pnpm --filter web dev # the demo page + /api/extract
pnpm --filter @receipt-evidence/mobile prebuild # generate ios/ + android/ (not committed)
node --test packages/contract/test/guards.test.ts # one file
node --test --test-name-pattern '<substring>' <file> # one test
node scripts/measure-corpus.mjs # re-derive the per-field baseline table
```

There is **no test framework**, deliberately. Node runs `.ts` directly by stripping types, so tests import `../src/x.ts` with the extension. Adding vitest or jest is a plan violation, not a preference.

Node strips types rather than checking them, so `pnpm test` says nothing about type errors — `pnpm typecheck` is the only type gate, and it has `strict`, `noUnusedLocals` and `noUnusedParameters` on.

## What the system does

A receipt's OCR text goes through a deterministic parser first; a model is asked only for what the parser could not derive; then deterministic code checks everything the model said.

`packages/contract` is the centre of gravity — the parser, the guards, the schema, and the types all live there so the server, the demo page and the mobile app share one definition of what a receipt fact is. Its tests need no network and no device.

`apps/web/src/extract.ts` is the pipeline. `apps/web/src/model-client.ts` is the only place that talks to OpenAI, and `extract()` takes the client as a parameter so tests substitute a fake — **no test in this repository makes a network call.**

`apps/mobile` is checked by its own `tsconfig.json` (React Native needs Expo's compiler settings), which the root `typecheck` script runs after the workspace one; `tsconfig.base.json` excludes it. Its testable logic lives in `src/capture.ts`, which imports the scanner for *types only* so `node --test` never loads React Native.

## Invariants that will bite

- **A value that fails a check is kept, marked `verified: false`, and listed in `unverified`.** Never dropped, never presented as fact. Dropping it hides the interesting half; presenting it is the failure this project exists to prevent. The same rule is why `arithmetic.agrees` has a third state: `null` means "nothing to compare", which is not `false`.
- **Evidence is a run of adjacent lines, at most `MAX_EVIDENCE_LINES` of them.** `verifyEvidence` (guards) and `anchorToLines` (anchor) answer the same question and both call `findLineRuns` in `normalize.ts`, so they cannot drift — a value that verified but could not be anchored would be unshowable. The anchor draws the rectangle enclosing every line of the run.
It was exactly one line until the first device capture: a Korean receipt prints an item's name, its barcode and its price on three lines, so the model quoted all three and a correct reading was rejected. Three things keep the relaxation from undoing the guard, and each has a test that fails when it is removed. The lines must be **adjacent**, so a label from the top cannot be joined to an amount from the bottom. The match must **begin in the run's first line**, or a run is just a later match with unquoted padding in front — which also makes an unambiguous excerpt look ambiguous and lose its box. And the **cap** is what stops a model quoting the whole page and having every value in it verify: that is the empty-excerpt failure in a longer coat.
Both still fail closed on an empty excerpt: every string contains `""`, so without that check the guard verifies everything. So does an ambiguous anchor — two runs mean no box, because a box on the wrong row is worse than none.
- **Two guards per value, and the second one is type-specific.** `verifyEvidence` asks whether the excerpt is a real line; then `excerptContainsAmount` (amounts, read through the parser's own `amountsOnLine` so guard and parser cannot disagree), `excerptContainsText` (strings), or a `parseDate` round-trip (dates) asks whether the value is actually stated there. Never run only the first: for four months the string and date paths did exactly that, and a fabricated merchant quoting any real line shipped as `verified: true`. Parser-derived fields run the same guards as model-derived ones — there is no exempt source.
- **`packages/contract/test/fixtures/receipts/expected.json` is ground truth and is never edited to match the parser.** It is byte-identical to its source in `due_back`. If the parser disagrees, either the port has a bug or the manifest is genuinely wrong — and the second one needs a human, not a commit.
- **Money is integer minor units** (`amountMinor`) — KRW whole won, USD cents. Never a float.
- **Dates are read back with local getters, never `toISOString()`.** The parser builds `new Date(y, m-1, d)` — local midnight — so `toISOString().slice(0,10)` reports the previous day in any positive-offset zone. That bug shipped once here, marked `verified: true`, on every Korean receipt.
- **The OpenAI model identifier is never written from memory.** It lives in `docs/notes/model-identifier.md` with the URL and date it was read from.
- Versions are pinned, not floated: `zod@4.4.3`, `openai@7.5.0`, `next@16.3.1`, `react@19.2.8` (web), `expo@57.0.15`, `expo-file-system@57.0.5`, `react-native@0.86.2` + `react@19.2.3` (mobile).
The mobile pins come from Expo, not from npm's `latest` — prebuild and autolinking are coupled to what the SDK was built against, and `react-native@0.87.0` (npm's latest) is newer than any Expo SDK supports. The `react-native`/`react` pair was read from `bundledNativeModules.json`; `expo` and `expo-file-system` were raised from 57.0.14/57.0.4 by `expo run:ios` itself during the first native build, which aligns the manifest as part of prebuild. Those are the versions the build that succeeded actually used, so they are the ones recorded — note the CLI prints `Updated package.json | no changes` while doing it, so check `git status` after a prebuild rather than trusting that line.
`pnpm peers check` reports one unmet peer (`react-dom@19.2.8` wants `^19.2.8`, sees mobile's 19.2.3); each app still links its own react, verified through `apps/*/node_modules/react`.

## The parser is a port, and the port is the point

`packages/contract/src/{evidence,dates,amounts,total,currency,items,analyze}.ts` are ported from `due_back/lib/due_back/service/receipt_analyzer.dart` — read-only, never modify it. Every ported file names its Dart source path and line range in a header comment.

The comments carry real edge cases (`TOTAL NUMBER OF ITEMS SOLD - 10` is not a total, `12,900원` is money but `원두커피` is coffee, a clock time is not an amount). Port statement by statement; do not paraphrase a condition into something that merely passes the listed tests.

Two deliberate deviations from the Dart, both documented at their site: the merchant skips lines that parse as an amount or a date (Dart takes `lines.first`, which publishes an amount as the merchant name on one corpus receipt), and Dart's `confidence` score is not ported — this system reports verification instead.

## Before trusting a test

Nine defects on this project were tests or gates that passed while proving nothing — a corpus gate that could not fail, a "pin" whose inputs missed the branch it named, a schema assertion that held with or without the setting it claimed to check. When you add a guard or a gate, break the thing it protects and watch it fail before you believe it.

## Current state

Every task in the plan is implemented: the parser and its corpus gate, the guards, arithmetic, anchoring, the schema, `/api/extract`, the demo page, the image fallback, the Expo app, and the README and CI. The suite and both typechecks pass; `pnpm test` reports the count, and no document restates it.

**One step is outstanding and it needs a human: Task 16 Step 4, the device pass.** The app has never been run — no `expo prebuild`, no native build, no camera or gallery capture on a real phone. It installs on the operator's daily iPhone, so it is theirs to authorise.

The branch `feat/scaffold-parser-and-extraction` has never been pushed; only `main` exists on the remote.
100 changes: 96 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,101 @@
Scan a receipt, get structured data that can show its work.

Every extracted value quotes the line of recognised text it came from, and a deterministic parser — not a second model — decides whether the model earned its answer.
A value that cannot show its evidence is reported as unverified rather than presented as fact.
A value that cannot show its evidence is kept and reported as unverified, never dropped and never presented as fact.

The design, including the measured baseline that decides where the model belongs, is in
[`docs/specs/2026-08-19-receipt-evidence-design.md`](docs/specs/2026-08-19-receipt-evidence-design.md).
## Why a model is here at all

Nothing is implemented yet.
A deterministic parser was ported first, then measured against 12 real anonymised receipts (6 Korean, 6 English) before any model was involved.
Re-derive this table any time with `node scripts/measure-corpus.mjs`; the long form is in [`docs/notes/corpus-baseline.md`](docs/notes/corpus-baseline.md).

| Field | Parser returned a value | …and it was right |
| ------------ | ----------------------- | ------------------ |
| currency | 12 / 12 | 12 / 12 |
| purchaseDate | 11 / 12 | 11 / 12 |
| merchant | 12 / 12 | 7 / 12 |
| paidTotal | 12 / 12 | 7 / 12 |
| reference | 1 / 12 | 1 / 12 |
| line items | 3 / 12 returned one | **0 / 12 correct** |

Two readings, and the second one is the point.

Currency is solved outright — it is not in the model's schema at all. The date nearly is: when the parser reads one, the request tells the model so and its answer would be discarded anyway, so it is asked only for the one receipt in twelve where the parser finds none.
Line items are where the parser has nothing — and worse, on 3 receipts it _invents_ one, reading a barcode fragment like `HE500* 100` as an item named `HE500*` priced at 100.
Merchant and paid total look complete and are wrong 5 times each, returning a garbled brand mark, or a barcode run picked up by the largest-amount fallback.

That last row is why this repository exists.
**A deterministic parser fails the same way a model does: confidently, with no signal attached.**
"Rule-based, therefore trustworthy" does not survive contact with real OCR.
So the answer is not to prefer one source over the other — it is to make every value, from either source, carry the line it was read from, and to check that line independently before calling it verified.

## How a value earns `verified: true`

1. **The parser goes first.** Whatever it derives is the baseline, and the model is never asked for it.
2. **The model is asked only for the gaps** — line items, plus any header field the parser left blank — and must quote a verbatim page excerpt for every value it reports.
3. **Deterministic code checks the reply.** The excerpt must appear on the page, within a run of at most three adjacent lines (`verifyEvidence`) — a receipt prints one item's name and price on separate rows, and the cap plus the adjacency rule are what stop a model quoting the page whole — and the value must actually be stated in that excerpt — an amount checked against the line's amounts as the parser itself reads them, a string as text, a date by re-parsing the line and comparing the day. Then the parser re-reads the cited line on its own and reports any disagreement, whatever the guard decided.
4. **The items are re-added independently** and compared with the claimed total. `agrees: null` is a third state — nothing to compare — and is shown as itself rather than folded into a failure.
5. **Anything that fails is kept**, marked `verified: false`, and listed in `unverified`. Dropping it would hide the interesting half.

Both guards fail closed on an empty excerpt, because every string contains `""` — without that check the guard would verify everything.
That one shipped here, and is now pinned by a test.

## Quick start

Requires Node 24+ and pnpm 11.

```bash
pnpm install
pnpm test # no network, no device
pnpm typecheck # the only type gate — node strips types, it does not check them
```

The demo page takes pasted OCR text (and optionally a receipt photo) and renders the result with its evidence.
Set `OPENAI_API_KEY` in the web app's local environment file, then:

```bash
pnpm --filter web dev # http://localhost:3000
```

## The mobile app

`apps/mobile` is the capture path: the platform document scanner, on-device OCR, one request, and evidence boxes drawn on the photo.

**Expo Go cannot load it.**
`react-native-receipt-scanner` is a native module, so the app needs a dev client:

```bash
pnpm --filter web dev # the API the app posts to
pnpm --filter @receipt-evidence/mobile prebuild # generates ios/ and android/
pnpm --filter @receipt-evidence/mobile ios
```

`ios/` and `android/` are generated rather than committed, so `prebuild` is the first step on a fresh clone.

No API URL to configure: the app derives it from the dev server it was loaded from, which is an address the device can already reach.
To point it somewhere else, set `EXPO_PUBLIC_API_URL` — but set it **where Metro runs**, not on the `ios` command:

```bash
EXPO_PUBLIC_API_URL=https://receipts.example.com pnpm --filter @receipt-evidence/mobile start
```

`EXPO_PUBLIC_*` variables are substituted into the source by Babel during Metro's transform, so the value has to be in Metro's environment. Putting it on `expo run:ios` reaches the native build and never the JS bundle whenever a dev server is already running — and `expo run:ios` prints `Skipping dev server` in exactly that case.

A page is sent as text alone when its OCR clears the scanner's floor; only a page below the floor also uploads its JPEG, so the image leaves the device exactly when the text cannot carry the work.

## Layout

```log
packages/contract/ the parser, the guards, the schema, the response types — shared by every client
apps/web/ POST /api/extract and the demo page
apps/mobile/ the Expo app
docs/ the design spec, the plan, and the measured baseline
scripts/ measure-corpus.mjs, which re-derives the table above
```

`packages/contract/src/{evidence,dates,amounts,total,currency,items,analyze}.ts` are a statement-by-statement port of a Dart receipt parser from a stopped project, kept honest by that project's own 12-receipt corpus — including the receipts it gets wrong, which are pinned by exact value so a change in behaviour fails loudly rather than passing silently.

There is no test framework, deliberately: Node runs TypeScript directly, so `node --test` is the whole harness.

## License

MIT — see [LICENSE](LICENSE).
5 changes: 5 additions & 0 deletions apps/mobile/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Generated by `expo prebuild` — this app uses Continuous Native Generation,
# so the native projects are rebuilt from app.json rather than committed.
/ios
/android
/.expo
Loading
Loading