diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..8b1e724 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,25 @@ +name: ci + +on: [push, pull_request] + +# The job only reads the repository; nothing here writes to it. +permissions: + contents: read + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + # checkout leaves its token in .git/config by default, where any + # later step (or a dependency's install script) could reuse it. + persist-credentials: false + - uses: pnpm/action-setup@v4 + - uses: actions/setup-node@v4 + with: + node-version: 24 + cache: pnpm + - run: pnpm install --frozen-lockfile + - run: pnpm test + - run: pnpm typecheck diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..f0eb565 --- /dev/null +++ b/.gitignore @@ -0,0 +1,9 @@ +node_modules/ +.next/ +next-env.d.ts +.expo/ +.superpowers/ +*.tsbuildinfo +.DS_Store +.env* +!.env.example diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..f0a42fb --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,67 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +Read `docs/specs/2026-08-19-receipt-evidence-design.md` for why this exists and what the pipeline is, `docs/notes/corpus-baseline.md` for the measured numbers the README quotes, and `docs/plans/2026-08-19-receipt-evidence.md` for the task-by-task build. +This file covers what those do not, or what will bite before you get to them. + +## Commands + +```bash +pnpm test # node --test over packages/*/test and apps/*/test +pnpm typecheck # tsc over the workspace, then apps/mobile's own tsconfig +pnpm --filter web dev # the demo page + /api/extract +pnpm --filter @receipt-evidence/mobile prebuild # generate ios/ + android/ (not committed) +node --test packages/contract/test/guards.test.ts # one file +node --test --test-name-pattern '' # one test +node scripts/measure-corpus.mjs # re-derive the per-field baseline table +``` + +There is **no test framework**, deliberately. Node runs `.ts` directly by stripping types, so tests import `../src/x.ts` with the extension. Adding vitest or jest is a plan violation, not a preference. + +Node strips types rather than checking them, so `pnpm test` says nothing about type errors — `pnpm typecheck` is the only type gate, and it has `strict`, `noUnusedLocals` and `noUnusedParameters` on. + +## What the system does + +A receipt's OCR text goes through a deterministic parser first; a model is asked only for what the parser could not derive; then deterministic code checks everything the model said. + +`packages/contract` is the centre of gravity — the parser, the guards, the schema, and the types all live there so the server, the demo page and the mobile app share one definition of what a receipt fact is. Its tests need no network and no device. + +`apps/web/src/extract.ts` is the pipeline. `apps/web/src/model-client.ts` is the only place that talks to OpenAI, and `extract()` takes the client as a parameter so tests substitute a fake — **no test in this repository makes a network call.** + +`apps/mobile` is checked by its own `tsconfig.json` (React Native needs Expo's compiler settings), which the root `typecheck` script runs after the workspace one; `tsconfig.base.json` excludes it. Its testable logic lives in `src/capture.ts`, which imports the scanner for *types only* so `node --test` never loads React Native. + +## Invariants that will bite + +- **A value that fails a check is kept, marked `verified: false`, and listed in `unverified`.** Never dropped, never presented as fact. Dropping it hides the interesting half; presenting it is the failure this project exists to prevent. The same rule is why `arithmetic.agrees` has a third state: `null` means "nothing to compare", which is not `false`. +- **Evidence is a run of adjacent lines, at most `MAX_EVIDENCE_LINES` of them.** `verifyEvidence` (guards) and `anchorToLines` (anchor) answer the same question and both call `findLineRuns` in `normalize.ts`, so they cannot drift — a value that verified but could not be anchored would be unshowable. The anchor draws the rectangle enclosing every line of the run. + It was exactly one line until the first device capture: a Korean receipt prints an item's name, its barcode and its price on three lines, so the model quoted all three and a correct reading was rejected. Three things keep the relaxation from undoing the guard, and each has a test that fails when it is removed. The lines must be **adjacent**, so a label from the top cannot be joined to an amount from the bottom. The match must **begin in the run's first line**, or a run is just a later match with unquoted padding in front — which also makes an unambiguous excerpt look ambiguous and lose its box. And the **cap** is what stops a model quoting the whole page and having every value in it verify: that is the empty-excerpt failure in a longer coat. + Both still fail closed on an empty excerpt: every string contains `""`, so without that check the guard verifies everything. So does an ambiguous anchor — two runs mean no box, because a box on the wrong row is worse than none. +- **Two guards per value, and the second one is type-specific.** `verifyEvidence` asks whether the excerpt is a real line; then `excerptContainsAmount` (amounts, read through the parser's own `amountsOnLine` so guard and parser cannot disagree), `excerptContainsText` (strings), or a `parseDate` round-trip (dates) asks whether the value is actually stated there. Never run only the first: for four months the string and date paths did exactly that, and a fabricated merchant quoting any real line shipped as `verified: true`. Parser-derived fields run the same guards as model-derived ones — there is no exempt source. +- **`packages/contract/test/fixtures/receipts/expected.json` is ground truth and is never edited to match the parser.** It is byte-identical to its source in `due_back`. If the parser disagrees, either the port has a bug or the manifest is genuinely wrong — and the second one needs a human, not a commit. +- **Money is integer minor units** (`amountMinor`) — KRW whole won, USD cents. Never a float. +- **Dates are read back with local getters, never `toISOString()`.** The parser builds `new Date(y, m-1, d)` — local midnight — so `toISOString().slice(0,10)` reports the previous day in any positive-offset zone. That bug shipped once here, marked `verified: true`, on every Korean receipt. +- **The OpenAI model identifier is never written from memory.** It lives in `docs/notes/model-identifier.md` with the URL and date it was read from. +- Versions are pinned, not floated: `zod@4.4.3`, `openai@7.5.0`, `next@16.3.1`, `react@19.2.8` (web), `expo@57.0.15`, `expo-file-system@57.0.5`, `react-native@0.86.2` + `react@19.2.3` (mobile). + The mobile pins come from Expo, not from npm's `latest` — prebuild and autolinking are coupled to what the SDK was built against, and `react-native@0.87.0` (npm's latest) is newer than any Expo SDK supports. The `react-native`/`react` pair was read from `bundledNativeModules.json`; `expo` and `expo-file-system` were raised from 57.0.14/57.0.4 by `expo run:ios` itself during the first native build, which aligns the manifest as part of prebuild. Those are the versions the build that succeeded actually used, so they are the ones recorded — note the CLI prints `Updated package.json | no changes` while doing it, so check `git status` after a prebuild rather than trusting that line. + `pnpm peers check` reports one unmet peer (`react-dom@19.2.8` wants `^19.2.8`, sees mobile's 19.2.3); each app still links its own react, verified through `apps/*/node_modules/react`. + +## The parser is a port, and the port is the point + +`packages/contract/src/{evidence,dates,amounts,total,currency,items,analyze}.ts` are ported from `due_back/lib/due_back/service/receipt_analyzer.dart` — read-only, never modify it. Every ported file names its Dart source path and line range in a header comment. + +The comments carry real edge cases (`TOTAL NUMBER OF ITEMS SOLD - 10` is not a total, `12,900원` is money but `원두커피` is coffee, a clock time is not an amount). Port statement by statement; do not paraphrase a condition into something that merely passes the listed tests. + +Two deliberate deviations from the Dart, both documented at their site: the merchant skips lines that parse as an amount or a date (Dart takes `lines.first`, which publishes an amount as the merchant name on one corpus receipt), and Dart's `confidence` score is not ported — this system reports verification instead. + +## Before trusting a test + +Nine defects on this project were tests or gates that passed while proving nothing — a corpus gate that could not fail, a "pin" whose inputs missed the branch it named, a schema assertion that held with or without the setting it claimed to check. When you add a guard or a gate, break the thing it protects and watch it fail before you believe it. + +## Current state + +Every task in the plan is implemented: the parser and its corpus gate, the guards, arithmetic, anchoring, the schema, `/api/extract`, the demo page, the image fallback, the Expo app, and the README and CI. The suite and both typechecks pass; `pnpm test` reports the count, and no document restates it. + +**One step is outstanding and it needs a human: Task 16 Step 4, the device pass.** The app has never been run — no `expo prebuild`, no native build, no camera or gallery capture on a real phone. It installs on the operator's daily iPhone, so it is theirs to authorise. + +The branch `feat/scaffold-parser-and-extraction` has never been pushed; only `main` exists on the remote. diff --git a/README.md b/README.md index a9c6ee9..2783fb6 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,101 @@ Scan a receipt, get structured data that can show its work. Every extracted value quotes the line of recognised text it came from, and a deterministic parser — not a second model — decides whether the model earned its answer. -A value that cannot show its evidence is reported as unverified rather than presented as fact. +A value that cannot show its evidence is kept and reported as unverified, never dropped and never presented as fact. -The design, including the measured baseline that decides where the model belongs, is in -[`docs/specs/2026-08-19-receipt-evidence-design.md`](docs/specs/2026-08-19-receipt-evidence-design.md). +## Why a model is here at all -Nothing is implemented yet. +A deterministic parser was ported first, then measured against 12 real anonymised receipts (6 Korean, 6 English) before any model was involved. +Re-derive this table any time with `node scripts/measure-corpus.mjs`; the long form is in [`docs/notes/corpus-baseline.md`](docs/notes/corpus-baseline.md). + +| Field | Parser returned a value | …and it was right | +| ------------ | ----------------------- | ------------------ | +| currency | 12 / 12 | 12 / 12 | +| purchaseDate | 11 / 12 | 11 / 12 | +| merchant | 12 / 12 | 7 / 12 | +| paidTotal | 12 / 12 | 7 / 12 | +| reference | 1 / 12 | 1 / 12 | +| line items | 3 / 12 returned one | **0 / 12 correct** | + +Two readings, and the second one is the point. + +Currency is solved outright — it is not in the model's schema at all. The date nearly is: when the parser reads one, the request tells the model so and its answer would be discarded anyway, so it is asked only for the one receipt in twelve where the parser finds none. +Line items are where the parser has nothing — and worse, on 3 receipts it _invents_ one, reading a barcode fragment like `HE500* 100` as an item named `HE500*` priced at 100. +Merchant and paid total look complete and are wrong 5 times each, returning a garbled brand mark, or a barcode run picked up by the largest-amount fallback. + +That last row is why this repository exists. +**A deterministic parser fails the same way a model does: confidently, with no signal attached.** +"Rule-based, therefore trustworthy" does not survive contact with real OCR. +So the answer is not to prefer one source over the other — it is to make every value, from either source, carry the line it was read from, and to check that line independently before calling it verified. + +## How a value earns `verified: true` + +1. **The parser goes first.** Whatever it derives is the baseline, and the model is never asked for it. +2. **The model is asked only for the gaps** — line items, plus any header field the parser left blank — and must quote a verbatim page excerpt for every value it reports. +3. **Deterministic code checks the reply.** The excerpt must appear on the page, within a run of at most three adjacent lines (`verifyEvidence`) — a receipt prints one item's name and price on separate rows, and the cap plus the adjacency rule are what stop a model quoting the page whole — and the value must actually be stated in that excerpt — an amount checked against the line's amounts as the parser itself reads them, a string as text, a date by re-parsing the line and comparing the day. Then the parser re-reads the cited line on its own and reports any disagreement, whatever the guard decided. +4. **The items are re-added independently** and compared with the claimed total. `agrees: null` is a third state — nothing to compare — and is shown as itself rather than folded into a failure. +5. **Anything that fails is kept**, marked `verified: false`, and listed in `unverified`. Dropping it would hide the interesting half. + +Both guards fail closed on an empty excerpt, because every string contains `""` — without that check the guard would verify everything. +That one shipped here, and is now pinned by a test. + +## Quick start + +Requires Node 24+ and pnpm 11. + +```bash +pnpm install +pnpm test # no network, no device +pnpm typecheck # the only type gate — node strips types, it does not check them +``` + +The demo page takes pasted OCR text (and optionally a receipt photo) and renders the result with its evidence. +Set `OPENAI_API_KEY` in the web app's local environment file, then: + +```bash +pnpm --filter web dev # http://localhost:3000 +``` + +## The mobile app + +`apps/mobile` is the capture path: the platform document scanner, on-device OCR, one request, and evidence boxes drawn on the photo. + +**Expo Go cannot load it.** +`react-native-receipt-scanner` is a native module, so the app needs a dev client: + +```bash +pnpm --filter web dev # the API the app posts to +pnpm --filter @receipt-evidence/mobile prebuild # generates ios/ and android/ +pnpm --filter @receipt-evidence/mobile ios +``` + +`ios/` and `android/` are generated rather than committed, so `prebuild` is the first step on a fresh clone. + +No API URL to configure: the app derives it from the dev server it was loaded from, which is an address the device can already reach. +To point it somewhere else, set `EXPO_PUBLIC_API_URL` — but set it **where Metro runs**, not on the `ios` command: + +```bash +EXPO_PUBLIC_API_URL=https://receipts.example.com pnpm --filter @receipt-evidence/mobile start +``` + +`EXPO_PUBLIC_*` variables are substituted into the source by Babel during Metro's transform, so the value has to be in Metro's environment. Putting it on `expo run:ios` reaches the native build and never the JS bundle whenever a dev server is already running — and `expo run:ios` prints `Skipping dev server` in exactly that case. + +A page is sent as text alone when its OCR clears the scanner's floor; only a page below the floor also uploads its JPEG, so the image leaves the device exactly when the text cannot carry the work. + +## Layout + +```log +packages/contract/ the parser, the guards, the schema, the response types — shared by every client +apps/web/ POST /api/extract and the demo page +apps/mobile/ the Expo app +docs/ the design spec, the plan, and the measured baseline +scripts/ measure-corpus.mjs, which re-derives the table above +``` + +`packages/contract/src/{evidence,dates,amounts,total,currency,items,analyze}.ts` are a statement-by-statement port of a Dart receipt parser from a stopped project, kept honest by that project's own 12-receipt corpus — including the receipts it gets wrong, which are pinned by exact value so a change in behaviour fails loudly rather than passing silently. + +There is no test framework, deliberately: Node runs TypeScript directly, so `node --test` is the whole harness. + +## License + +MIT — see [LICENSE](LICENSE). diff --git a/apps/mobile/.gitignore b/apps/mobile/.gitignore new file mode 100644 index 0000000..34ba45e --- /dev/null +++ b/apps/mobile/.gitignore @@ -0,0 +1,5 @@ +# Generated by `expo prebuild` — this app uses Continuous Native Generation, +# so the native projects are rebuilt from app.json rather than committed. +/ios +/android +/.expo diff --git a/apps/mobile/App.tsx b/apps/mobile/App.tsx new file mode 100644 index 0000000..470c15d --- /dev/null +++ b/apps/mobile/App.tsx @@ -0,0 +1,240 @@ +// The capture path this sample is actually about: scan a receipt, decide +// per page whether its text can travel alone, post one request, and show +// every extracted value beside the pixels it was read from. +import { useState } from "react"; +import { ActivityIndicator, Button, Image, NativeModules, ScrollView, StyleSheet, Text, View } from "react-native"; +import { StatusBar } from "expo-status-bar"; +import { File } from "expo-file-system"; +import { scan, DEFAULT_OCR_FLOOR } from "react-native-receipt-scanner"; +import type { ReceiptImage } from "react-native-receipt-scanner"; +import type { ExtractedField, ExtractedItem, ExtractionResponse, Frame, Page } from "@receipt-evidence/contract/response"; +import { apiBaseUrl } from "./src/api.ts"; +import { clearsFloor } from "./src/capture.ts"; +import { EvidenceOverlay } from "./src/EvidenceOverlay.tsx"; + +// Derived from the dev server this bundle came from, so it points at the +// machine running `pnpm --filter web dev` without being configured. See +// src/api.ts for why the env var alone was not enough. +// `SourceCode.scriptURL` is where this bundle was loaded from — the one host +// the running app has already proved it can reach. See src/api.ts for why +// Constants.expoConfig.hostUri, the obvious choice, is empty in this app. +const scriptUrl = (NativeModules["SourceCode"] as { getConstants?: () => { scriptURL?: string } } | undefined) + ?.getConstants?.() + ?.scriptURL; +const API_URL = apiBaseUrl(process.env.EXPO_PUBLIC_API_URL, scriptUrl); + +/** The page whose photo the result view shows. Every other page's values are + * still listed — only their boxes have nowhere to be drawn. */ +const PRIMARY_PAGE = 0; + +type Status = + | { kind: "idle" } + | { kind: "working"; step: string } + | { kind: "failed"; message: string } + | { kind: "done"; image: ReceiptImage; result: ExtractionResponse }; + +/** Turns one capture into a request page. Text always; the JPEG only when + * the text cannot carry the work — that decision, not the upload, is what + * the OCR floor is for. */ +async function toPage(image: ReceiptImage): Promise { + const page: Page = { text: image.ocrText ?? "", lines: image.ocrLines ?? [] }; + if (clearsFloor(image.ocrQuality, DEFAULT_OCR_FLOOR)) return page; + const base64 = await new File(image.uri).base64(); + // mimeType is always "image/jpeg" on this package, but reading it from the + // capture keeps the media type travelling with the bytes instead of being + // asserted here — the server hands the whole data URL straight to the model. + return { ...page, imageDataUrl: `data:${image.mimeType};base64,${base64}` }; +} + +/** Boxes for ONE page, in that page's own pixel space. + * + * The pageIndex filter is not optional: a scan may carry up to three pages, + * each anchored against its own OCR geometry, and the view shows one photo. + * Without it a box computed on page 1 was painted on page 0's image, pointing + * the reader at unrelated text — the worst failure available to an app whose + * claim is that a value is shown beside the pixels it was read from. */ +function boxesOf(result: ExtractionResponse, pageIndex: number, wanted: boolean): Frame[] { + const entries = [ + ...Object.values(result.fields).filter( + (field): field is ExtractedField | ExtractedField => + typeof field === "object" && field !== null, + ), + ...result.items, + ]; + return entries + .filter( + (entry) => + entry.evidence.pageIndex === pageIndex && entry.verified === wanted && entry.evidence.box !== null, + ) + .map((entry) => entry.evidence.box as Frame); +} + +export default function App() { + const [status, setStatus] = useState({ kind: "idle" }); + + async function capture(source: "camera" | "gallery") { + try { + setStatus({ kind: "working", step: "scanning" }); + const scanned = await scan({ + source, + maxPages: 3, + // Geometry is off by default, and without it there is nothing to draw + // a box from. The floor is disabled here on purpose: the package's own + // gate DROPS a below-floor capture into `rejectedImages`, and that is + // exactly the capture whose image most needs to be sent. This app + // keeps every page and decides per page what it carries (`toPage`). + ocrGeometry: true, + ocrFloor: false, + }); + if (scanned.status === "cancelled") return setStatus({ kind: "idle" }); + const [primary] = scanned.images; + if (primary === undefined) { + return setStatus({ kind: "failed", message: "the scanner returned no page" }); + } + + setStatus({ kind: "working", step: "extracting" }); + const pages = await Promise.all(scanned.images.map(toPage)); + // The OCR text is the pipeline's actual input, and every defect found on + // a real receipt so far has turned on its exact line structure — where + // the printer spaced a label, whether OCR kept `barcode 1 2,000` on one + // line or split it into three. Without it a capture can only be argued + // about from a screenshot. Dev builds only, because it is the receipt's + // contents. + if (__DEV__) { + for (const [index, page] of pages.entries()) { + console.log(`[receipt-evidence] page ${index} ocrText:\n${page.text}`); + } + } + const response = await fetch(`${API_URL}/api/extract`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ pages }), + }); + const body: unknown = await response.json(); + if (!response.ok) { + const message = + typeof body === "object" && body !== null && "error" in body + ? String((body as { error: unknown }).error) + : response.statusText; + return setStatus({ kind: "failed", message }); + } + setStatus({ kind: "done", image: primary, result: body as ExtractionResponse }); + } catch (error) { + setStatus({ kind: "failed", message: error instanceof Error ? error.message : String(error) }); + } + } + + return ( + + + Receipt Evidence + {API_URL} + +