-
Notifications
You must be signed in to change notification settings - Fork 0
feat: receipt extraction that shows its evidence #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
69 commits
Select commit
Hold shift + click to select a range
f6c84dc
chore: 🔧 ignore build output, env files, and agent scratch
AndrewDongminYoo 34e3081
build: 🏗️ set up the pnpm workspace and the contract package
AndrewDongminYoo 77de059
fix: 🔧 add types and allowImportingTsExtensions to tsconfig for type …
AndrewDongminYoo bf1c177
feat(contract): ✨ split OCR text into indexed evidence lines
AndrewDongminYoo 2b16c0b
docs(plans): 📝 correct Task 2's line-index semantics to match the Dar…
AndrewDongminYoo 667f2fd
fix(contract): evidence lines index counts recognised lines only
AndrewDongminYoo e1b332b
fix(contract): correct OcrEvidence JSDoc to match implementation
AndrewDongminYoo 2398cb9
feat(contract): ✨ parse receipt dates and reject expiries
AndrewDongminYoo 7f7cf5c
docs(plans): 📝 make Task 3 read every date match, as the Dart source …
AndrewDongminYoo dd95224
fix(contract): 🐛 parseDate skips calendar-invalid matches like Dart's…
AndrewDongminYoo 19ef22f
feat(contract): ✨ read receipt amounts as integer minor units
AndrewDongminYoo 5acfdcf
docs(plans): 📝 cite the Dart range where amount parsing actually lives
AndrewDongminYoo 2cdba4d
docs(contract): 📝 document DATE_PATTERN_G reuse hazard and minorUnits…
AndrewDongminYoo e59b856
docs(plans): 📝 stop asserting a fare row becomes the selected total
AndrewDongminYoo 5fb53b8
feat(contract): ✨ pick the paid total apart from discounts and counts
AndrewDongminYoo 18304b8
feat(contract): ✨ infer the receipt currency from its markers
AndrewDongminYoo 2262d67
fix(contract): 🔧 drop dead import, tighten typecheck to catch unused …
AndrewDongminYoo 617bfd0
test(contract): 🐛 pin the _isPricedItem gap with a test that can fail
AndrewDongminYoo 8f6182b
feat(contract): ✨ extract priced line items with their evidence
AndrewDongminYoo df3b958
fix(contract): 🔧 close the isPricedItem gap in cents-row currency inf…
AndrewDongminYoo e5ee163
fix(contract): 🐛 correct items.ts source citations
AndrewDongminYoo a86aca5
feat(contract): ✨ assemble the deterministic receipt parser
AndrewDongminYoo 53e93af
fix(contract): 🎯 skip amount/date lines when selecting the merchant
AndrewDongminYoo 09f97b6
docs(contract): 📝 correct REFERENCE_LABEL duplication comment in anal…
AndrewDongminYoo f29c18b
test(contract): ✅ gate the parser on the 12-receipt corpus
AndrewDongminYoo a2e2e88
docs: 📝 replace the claimed parser baseline with the measured one
AndrewDongminYoo bdec458
fix(contract): 🎯 pin items divergence, fix KR-04 merchant sign, close…
AndrewDongminYoo 570379f
test(contract): 🔒 pin exact values for non-derivable merchant/paidTot…
AndrewDongminYoo 4fd2142
feat(contract): ✨ verify model values against their evidence
AndrewDongminYoo fe574e5
docs(plans): 📝 make the guard's flagship test exercise both halves
AndrewDongminYoo 40746ca
test(contract): pin the excerpt-real-but-value-wrong hallucination case
AndrewDongminYoo f0c6dc8
fix(contract): match evidence per line, not across the whole page
AndrewDongminYoo f6ffe13
fix(contract): verifyEvidence rejects an empty or whitespace-only exc…
AndrewDongminYoo 517d24b
feat(contract): ✨ recompute the item sum against the claimed total
AndrewDongminYoo ecd8218
feat(contract): ✨ anchor an evidence excerpt to its OCR box
AndrewDongminYoo 6cfc70a
refactor(contract): extract shared normalize function
AndrewDongminYoo 81cbd86
feat(contract): ✨ define the model reply schema once, in zod
AndrewDongminYoo 89e7da7
docs(plans): 📝 drop the currency branch and make the strictness test …
AndrewDongminYoo 0b257ba
fix(contract): drop dead currency field, make quantity optional, pin …
AndrewDongminYoo 75222a4
feat(web): ✨ extract a receipt through the parser, the model, and the…
AndrewDongminYoo b83c0a2
fix(web): report a rejected model reply instead of vanishing it into …
AndrewDongminYoo 258d54b
test(web): assert toStrictSchema against the real emitted schema, rec…
AndrewDongminYoo 11660cf
fix(web): read purchaseDate back through local getters, not toISOString
AndrewDongminYoo 23d64c0
feat(web): ✨ show extraction results with their evidence
AndrewDongminYoo 8b79fdd
docs(plans): 📝 add the image fallback as a task, since nothing implem…
AndrewDongminYoo 378dc3e
docs: 📝 add CLAUDE.md for future sessions in this repository
AndrewDongminYoo 36dc41b
feat(web): ✨ send a page's image when the caller attached one
AndrewDongminYoo 4a5ac00
refactor(contract): 🚚 move the response types where both clients can …
AndrewDongminYoo a39ed70
feat(mobile): ✨ scan a receipt and show its evidence on the image
AndrewDongminYoo 09cea88
docs: 📝 lead the README with the measured parser baseline
AndrewDongminYoo 1778835
fix(web): 🐛 answer a malformed request with 400, not an unhandled 500
AndrewDongminYoo a5d8c9a
docs: 📝 stop restating the test count in the README
AndrewDongminYoo 76b4d3f
fix(contract): 🐛 check what a value claims, not just where it was quo…
AndrewDongminYoo fe4ba18
fix(web): 🐛 stop uploading the receipt photo without being asked
AndrewDongminYoo 06660d6
docs: 📝 correct every document that named the removed guard
AndrewDongminYoo 5daa09d
build(mobile): ⬆️ adopt the versions the first native build actually …
AndrewDongminYoo 03a918a
fix(mobile): 🐛 derive the API host from the dev server, not from loca…
AndrewDongminYoo 5a13d17
fix(contract): 🐛 verify every part of a claim, and stop a gate that c…
AndrewDongminYoo ccdd896
fix(web): 🐛 read the whole scan as one document, and stop the server'…
AndrewDongminYoo 006c635
fix(web): 🐛 close the consent, image and error-label holes at the bou…
AndrewDongminYoo 69f4304
fix(contract): 🐛 refuse an amount JS cannot hold, and a box that coul…
AndrewDongminYoo 11588b0
docs: 📝 close the drift the review found between docs, config and code
AndrewDongminYoo c1183ac
fix(web): 🐛 let only the newest file selection write the image
AndrewDongminYoo ea9d221
fix(web): 🐛 judge the request before the server's own configuration
AndrewDongminYoo 4714afc
fix(mobile): 🐛 take the API host from the bundle URL, not from a mani…
AndrewDongminYoo 7a26d97
fix(contract): 🐛 read a Korean label whose printer spaced its characters
AndrewDongminYoo 5b06eb1
feat(contract): ✨ let evidence span the adjacent lines a receipt prin…
AndrewDongminYoo f1b4b3a
fix(contract): ✨ raise the evidence cap to the four lines a real rece…
AndrewDongminYoo 0a084d4
fix(contract): 🐛 stop a receipt's VAT shipping as its paid total
AndrewDongminYoo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| name: ci | ||
|
|
||
| on: [push, pull_request] | ||
|
|
||
| # The job only reads the repository; nothing here writes to it. | ||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| test: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| # checkout leaves its token in .git/config by default, where any | ||
| # later step (or a dependency's install script) could reuse it. | ||
| persist-credentials: false | ||
| - uses: pnpm/action-setup@v4 | ||
| - uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 24 | ||
| cache: pnpm | ||
| - run: pnpm install --frozen-lockfile | ||
| - run: pnpm test | ||
| - run: pnpm typecheck | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| node_modules/ | ||
| .next/ | ||
| next-env.d.ts | ||
| .expo/ | ||
| .superpowers/ | ||
| *.tsbuildinfo | ||
| .DS_Store | ||
| .env* | ||
| !.env.example |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # CLAUDE.md | ||
|
|
||
| This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. | ||
|
|
||
| Read `docs/specs/2026-08-19-receipt-evidence-design.md` for why this exists and what the pipeline is, `docs/notes/corpus-baseline.md` for the measured numbers the README quotes, and `docs/plans/2026-08-19-receipt-evidence.md` for the task-by-task build. | ||
| This file covers what those do not, or what will bite before you get to them. | ||
|
|
||
| ## Commands | ||
|
|
||
| ```bash | ||
| pnpm test # node --test over packages/*/test and apps/*/test | ||
| pnpm typecheck # tsc over the workspace, then apps/mobile's own tsconfig | ||
| pnpm --filter web dev # the demo page + /api/extract | ||
| pnpm --filter @receipt-evidence/mobile prebuild # generate ios/ + android/ (not committed) | ||
| node --test packages/contract/test/guards.test.ts # one file | ||
| node --test --test-name-pattern '<substring>' <file> # one test | ||
| node scripts/measure-corpus.mjs # re-derive the per-field baseline table | ||
| ``` | ||
|
|
||
| There is **no test framework**, deliberately. Node runs `.ts` directly by stripping types, so tests import `../src/x.ts` with the extension. Adding vitest or jest is a plan violation, not a preference. | ||
|
|
||
| Node strips types rather than checking them, so `pnpm test` says nothing about type errors — `pnpm typecheck` is the only type gate, and it has `strict`, `noUnusedLocals` and `noUnusedParameters` on. | ||
|
|
||
| ## What the system does | ||
|
|
||
| A receipt's OCR text goes through a deterministic parser first; a model is asked only for what the parser could not derive; then deterministic code checks everything the model said. | ||
|
|
||
| `packages/contract` is the centre of gravity — the parser, the guards, the schema, and the types all live there so the server, the demo page and the mobile app share one definition of what a receipt fact is. Its tests need no network and no device. | ||
|
|
||
| `apps/web/src/extract.ts` is the pipeline. `apps/web/src/model-client.ts` is the only place that talks to OpenAI, and `extract()` takes the client as a parameter so tests substitute a fake — **no test in this repository makes a network call.** | ||
|
|
||
| `apps/mobile` is checked by its own `tsconfig.json` (React Native needs Expo's compiler settings), which the root `typecheck` script runs after the workspace one; `tsconfig.base.json` excludes it. Its testable logic lives in `src/capture.ts`, which imports the scanner for *types only* so `node --test` never loads React Native. | ||
|
|
||
| ## Invariants that will bite | ||
|
|
||
| - **A value that fails a check is kept, marked `verified: false`, and listed in `unverified`.** Never dropped, never presented as fact. Dropping it hides the interesting half; presenting it is the failure this project exists to prevent. The same rule is why `arithmetic.agrees` has a third state: `null` means "nothing to compare", which is not `false`. | ||
| - **Evidence is a run of adjacent lines, at most `MAX_EVIDENCE_LINES` of them.** `verifyEvidence` (guards) and `anchorToLines` (anchor) answer the same question and both call `findLineRuns` in `normalize.ts`, so they cannot drift — a value that verified but could not be anchored would be unshowable. The anchor draws the rectangle enclosing every line of the run. | ||
| It was exactly one line until the first device capture: a Korean receipt prints an item's name, its barcode and its price on three lines, so the model quoted all three and a correct reading was rejected. Three things keep the relaxation from undoing the guard, and each has a test that fails when it is removed. The lines must be **adjacent**, so a label from the top cannot be joined to an amount from the bottom. The match must **begin in the run's first line**, or a run is just a later match with unquoted padding in front — which also makes an unambiguous excerpt look ambiguous and lose its box. And the **cap** is what stops a model quoting the whole page and having every value in it verify: that is the empty-excerpt failure in a longer coat. | ||
| Both still fail closed on an empty excerpt: every string contains `""`, so without that check the guard verifies everything. So does an ambiguous anchor — two runs mean no box, because a box on the wrong row is worse than none. | ||
| - **Two guards per value, and the second one is type-specific.** `verifyEvidence` asks whether the excerpt is a real line; then `excerptContainsAmount` (amounts, read through the parser's own `amountsOnLine` so guard and parser cannot disagree), `excerptContainsText` (strings), or a `parseDate` round-trip (dates) asks whether the value is actually stated there. Never run only the first: for four months the string and date paths did exactly that, and a fabricated merchant quoting any real line shipped as `verified: true`. Parser-derived fields run the same guards as model-derived ones — there is no exempt source. | ||
| - **`packages/contract/test/fixtures/receipts/expected.json` is ground truth and is never edited to match the parser.** It is byte-identical to its source in `due_back`. If the parser disagrees, either the port has a bug or the manifest is genuinely wrong — and the second one needs a human, not a commit. | ||
| - **Money is integer minor units** (`amountMinor`) — KRW whole won, USD cents. Never a float. | ||
| - **Dates are read back with local getters, never `toISOString()`.** The parser builds `new Date(y, m-1, d)` — local midnight — so `toISOString().slice(0,10)` reports the previous day in any positive-offset zone. That bug shipped once here, marked `verified: true`, on every Korean receipt. | ||
| - **The OpenAI model identifier is never written from memory.** It lives in `docs/notes/model-identifier.md` with the URL and date it was read from. | ||
| - Versions are pinned, not floated: `zod@4.4.3`, `openai@7.5.0`, `next@16.3.1`, `react@19.2.8` (web), `expo@57.0.15`, `expo-file-system@57.0.5`, `react-native@0.86.2` + `react@19.2.3` (mobile). | ||
| The mobile pins come from Expo, not from npm's `latest` — prebuild and autolinking are coupled to what the SDK was built against, and `react-native@0.87.0` (npm's latest) is newer than any Expo SDK supports. The `react-native`/`react` pair was read from `bundledNativeModules.json`; `expo` and `expo-file-system` were raised from 57.0.14/57.0.4 by `expo run:ios` itself during the first native build, which aligns the manifest as part of prebuild. Those are the versions the build that succeeded actually used, so they are the ones recorded — note the CLI prints `Updated package.json | no changes` while doing it, so check `git status` after a prebuild rather than trusting that line. | ||
| `pnpm peers check` reports one unmet peer (`react-dom@19.2.8` wants `^19.2.8`, sees mobile's 19.2.3); each app still links its own react, verified through `apps/*/node_modules/react`. | ||
|
|
||
| ## The parser is a port, and the port is the point | ||
|
|
||
| `packages/contract/src/{evidence,dates,amounts,total,currency,items,analyze}.ts` are ported from `due_back/lib/due_back/service/receipt_analyzer.dart` — read-only, never modify it. Every ported file names its Dart source path and line range in a header comment. | ||
|
|
||
| The comments carry real edge cases (`TOTAL NUMBER OF ITEMS SOLD - 10` is not a total, `12,900원` is money but `원두커피` is coffee, a clock time is not an amount). Port statement by statement; do not paraphrase a condition into something that merely passes the listed tests. | ||
|
|
||
| Two deliberate deviations from the Dart, both documented at their site: the merchant skips lines that parse as an amount or a date (Dart takes `lines.first`, which publishes an amount as the merchant name on one corpus receipt), and Dart's `confidence` score is not ported — this system reports verification instead. | ||
|
|
||
| ## Before trusting a test | ||
|
|
||
| Nine defects on this project were tests or gates that passed while proving nothing — a corpus gate that could not fail, a "pin" whose inputs missed the branch it named, a schema assertion that held with or without the setting it claimed to check. When you add a guard or a gate, break the thing it protects and watch it fail before you believe it. | ||
|
|
||
| ## Current state | ||
|
|
||
| Every task in the plan is implemented: the parser and its corpus gate, the guards, arithmetic, anchoring, the schema, `/api/extract`, the demo page, the image fallback, the Expo app, and the README and CI. The suite and both typechecks pass; `pnpm test` reports the count, and no document restates it. | ||
|
|
||
| **One step is outstanding and it needs a human: Task 16 Step 4, the device pass.** The app has never been run — no `expo prebuild`, no native build, no camera or gallery capture on a real phone. It installs on the operator's daily iPhone, so it is theirs to authorise. | ||
|
|
||
| The branch `feat/scaffold-parser-and-extraction` has never been pushed; only `main` exists on the remote. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # Generated by `expo prebuild` — this app uses Continuous Native Generation, | ||
| # so the native projects are rebuilt from app.json rather than committed. | ||
| /ios | ||
| /android | ||
| /.expo |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.