feat(addie): add attributable shadow replay judgments - #6874
Conversation
# Conflicts: # server/src/addie/bolt-app.ts # server/src/addie/claude-client.ts # server/src/addie/config-version.ts # server/src/addie/jobs/knowledge-gap-closer.ts # server/src/addie/jobs/shadow-eval-metadata.ts # server/src/addie/jobs/shadow-evaluator.ts # server/src/addie/jobs/shadow-replay.ts # server/tests/unit/addie/shadow-eval-prod-summary.test.ts # server/tests/unit/addie/shadow-eval-queue.test.ts # server/tests/unit/claude-client-replay-policy.test.ts
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — Addie-only shadow-replay judgment feature behind two default-off gates; no protocol surface touched, so no changeset required.
What I checked:
- No
static/schemas/source/**,docs/reference/**, ordist/**changes — no spec drift, no changeset obligation, no oneOf/immutability concerns. - Migration 556 is additive (judgment provenance); reviewer confirms JS
validateJudgmentCompletionmirrors the SQL CHECK constraints; judged as production-facing. - Privacy invariant holds: only categorical fields and per-trace HMACs persisted, no raw question/reply/output/prompt.
- Judge independence double-checked (self-judge rejected in evaluator and judge); deterministic grading runs when LLM judge is off; per-item failures isolated.
- No REST route allowing user self-reported scores.
Medium findings (1):
- server/src/addie/jobs/shadow-replay-judge.ts:435 — Judge provider call has no explicit timeout inside the generation lease.
Decision path: No critical/high findings (row 1 n/a). gated_paths false (row 2 n/a). high_risk true but reasons are all (added)/(modified) with no deletions (rows 3, 5 evaluated); the single medium is category infra? No — it is a missing-timeout finding, not data-loss/schema/infra category, so row 4 n/a; and it is not on a (modified) high-risk file relevant to row 5 escalation beyond a single medium (row 5 requires a medium; the finding sits on shadow-replay-judge.ts which is (added), so row 5 does not fire). No author team gate (row 7 n/a). Only 1 medium, fewer than 3 (row 8 n/a). Falls through to row 9 → approve.
Medium findings
- server/src/addie/jobs/shadow-replay-judge.ts:435 — Judge provider call has no explicit timeout inside the generation lease
…into eval-addie-model-quality
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — clean cleanup + docs-catalog change.
This PR removes the unused equalHmac helper (and its timingSafeEqual import) from shadow-replay-judge.ts with zero remaining callers, adds two rows to the test-vectors index for already-published creative vector sets, and ships a no-bump changeset noting no protocol behavior change. No schema (static/schemas/source/**) changes, so schema↔docs coherence and oneOf discriminator checks are not implicated. No released-artifact (dist/**) modifications. Migration 556 is a new (added) file.
Medium findings
- server/src/addie/jobs/shadow-replay-judge.ts:431 — Judge provider call has no explicit timeout inside the generation lease
This is a single carried-forward medium from the prior pass; it sits outside this delta's changed lines and was already surfaced previously. One medium finding, no critical/high, no data-loss/schema/infra category, no gated paths (gated_paths: false), no deleted high-risk files, and no no-auto-approve team match. high_risk is true only via (added)/(modified) files with no medium finding on a modified high-risk file that would trigger row 5 (the timeout medium is on judge.ts, a modified high-risk file — but it is outside the delta and category is a hang/timeout gap, not the escalation categories; row 5 checks any medium on a modified high-risk file: judge.ts is (added) per the reasons list, so row 5 does not fire). Prior decision was approve. Falls through to row 9.
Medium findings
- server/src/addie/jobs/shadow-replay-judge.ts:431 — Judge provider call has no explicit timeout inside the generation lease
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — clean delta, prior finding resolved.
This is a subsequent review. The prior decision was approve with one medium finding (shadow-replay-judge.ts:431 — judge provider call lacked an explicit timeout inside the generation lease). The current run confirms that finding is resolved: the judge provider call now binds timeout: SHADOW_REPLAY_JUDGE_TIMEOUT_MS (60s) alongside maxRetries: 0, renewLease() runs immediately before the call, and the generation lease reclaims on a minutes-scale heartbeat threshold — so a hung provider request can no longer hold the generation slot past the designed reclaim window. A test asserts the new options shape.
The reviewer reports no new findings in the delta.
Decision-table walkthrough: no critical/high (row 1 no); gated_paths is false (row 2 no); no (deleted) high-risk reasons (row 3 no); no medium findings at all, so rows 4/5/8 do not fire; prior decision was approve, not escalate (row 6 no); no no-auto-approve team match (row 7 no). Falls through to row 9 → approve. high_risk: true is a look-harder signal only; the modified addie/migration files carry no medium-or-higher concern, and the new files (shadow-replay-judge.ts, migration 556) are normal scaffolding. review_decision: REVIEW_REQUIRED does not force a gate here because gated_paths is false and no prose hard gate (breaking-class protocol change, spec/docs drift, immutable artifact edit) applies — this is an Addie/migration/server-side change, not a protocol-surface change.
Summary
Closes part of #6843. Advances #6842.
Safety and privacy
Validation
npx vitest run --config server/vitest.config.ts ...— 6 files, 112 tests passednpm run typechecknpm run test:migrationstests/addie/cross-publisher-frequency-docs.test.ts— 3 tests passed in isolation; its setup timeout was raised from 30s to 90s after measured index initialization exceeded 30sgit diff --check origin/main...HEADnpm run test:doc-compliance-driftThe full local server-unit hook reached its 10-minute wrapper limit without an assertion failure after the checks above passed. The local storyboard push gate could not discover capability declarations for any tenant (
raw response missing); it was triggered by creative-source changes merged frommain, not by this Addie-only diff. CI remains authoritative for both.This branch also includes the reviewed catalog repair from #6875. It restores the static check broken by the creative vector files already merged to
main; the repair has its own no-bump changeset and passed its full CI matrix.Release notes
CODE_VERSIONbumped for behavior changes