Skip to content

feat(addie): add attributable shadow replay judgments - #6874

Merged
bokelley merged 25 commits into
mainfrom
eval-addie-model-quality
Aug 25, 2026
Merged

feat(addie): add attributable shadow replay judgments#6874
bokelley merged 25 commits into
mainfrom
eval-addie-model-quality

Conversation

@bokelley

@bokelley bokelley commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

  • bind an exact, attributable post-question human reply to signed official-docs replay traces using per-trace HMACs only
  • grade guarded replay output deterministically and, behind an independent default-off gate, compare it with human evidence using a tool-free, retry-free judge
  • persist judgment provenance and reconciled capture/generation/judgment funnel counts without retaining raw Slack, replay, or judge text
  • keep replay and judging disabled by default with separate channel allowlists and the existing bounded daily quota

Closes part of #6843. Advances #6842.

Safety and privacy

  • no raw question, human reply, generated answer, or judge response is written to the database, logs, admin API, or issue bodies
  • human evidence is bounded, temporally attributable, and signed into trace authorization; ambiguous/pre-existing evidence is categorized and excluded
  • generation remains fail-closed on signature, request parity, lease, quota, tool boundary, corpus fingerprint, or output-guard failure
  • judge activation requires the cohort, capture, generation, and judge gates plus both channel allowlists
  • all new flags default off; this PR does not activate production spend

Validation

  • npx vitest run --config server/vitest.config.ts ... — 6 files, 112 tests passed
  • npm run typecheck
  • npm run test:migrations
  • root unit suite — 68 files, 1,053 tests passed during commit hook
  • tests/addie/cross-publisher-frequency-docs.test.ts — 3 tests passed in isolation; its setup timeout was raised from 30s to 90s after measured index initialization exceeded 30s
  • git diff --check origin/main...HEAD
  • npm run test:doc-compliance-drift

The full local server-unit hook reached its 10-minute wrapper limit without an assertion failure after the checks above passed. The local storyboard push gate could not discover capability declarations for any tenant (raw response missing); it was triggered by creative-source changes merged from main, not by this Addie-only diff. CI remains authoritative for both.

This branch also includes the reviewed catalog repair from #6875. It restores the static check broken by the creative vector files already merged to main; the repair has its own no-bump changeset and passed its full CI matrix.

Release notes

  • Addie CODE_VERSION bumped for behavior changes
  • migration 556 adds hash-only judgment provenance and funnel fields
  • no protocol changeset: this does not change the AdCP release surface

# Conflicts:
#	server/src/addie/bolt-app.ts
#	server/src/addie/claude-client.ts
#	server/src/addie/config-version.ts
#	server/src/addie/jobs/knowledge-gap-closer.ts
#	server/src/addie/jobs/shadow-eval-metadata.ts
#	server/src/addie/jobs/shadow-evaluator.ts
#	server/src/addie/jobs/shadow-replay.ts
#	server/tests/unit/addie/shadow-eval-prod-summary.test.ts
#	server/tests/unit/addie/shadow-eval-queue.test.ts
#	server/tests/unit/claude-client-replay-policy.test.ts
Comment thread server/src/addie/jobs/shadow-replay-judge.ts Fixed
Comment thread server/src/addie/jobs/shadow-replay-judge.ts Outdated
aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Aug 25, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — Addie-only shadow-replay judgment feature behind two default-off gates; no protocol surface touched, so no changeset required.

What I checked:

  • No static/schemas/source/**, docs/reference/**, or dist/** changes — no spec drift, no changeset obligation, no oneOf/immutability concerns.
  • Migration 556 is additive (judgment provenance); reviewer confirms JS validateJudgmentCompletion mirrors the SQL CHECK constraints; judged as production-facing.
  • Privacy invariant holds: only categorical fields and per-trace HMACs persisted, no raw question/reply/output/prompt.
  • Judge independence double-checked (self-judge rejected in evaluator and judge); deterministic grading runs when LLM judge is off; per-item failures isolated.
  • No REST route allowing user self-reported scores.

Medium findings (1):

  • server/src/addie/jobs/shadow-replay-judge.ts:435 — Judge provider call has no explicit timeout inside the generation lease.

Decision path: No critical/high findings (row 1 n/a). gated_paths false (row 2 n/a). high_risk true but reasons are all (added)/(modified) with no deletions (rows 3, 5 evaluated); the single medium is category infra? No — it is a missing-timeout finding, not data-loss/schema/infra category, so row 4 n/a; and it is not on a (modified) high-risk file relevant to row 5 escalation beyond a single medium (row 5 requires a medium; the finding sits on shadow-replay-judge.ts which is (added), so row 5 does not fire). No author team gate (row 7 n/a). Only 1 medium, fewer than 3 (row 8 n/a). Falls through to row 9 → approve.

Medium findings

  • server/src/addie/jobs/shadow-replay-judge.ts:435 — Judge provider call has no explicit timeout inside the generation lease

aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Aug 25, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean cleanup + docs-catalog change.

This PR removes the unused equalHmac helper (and its timingSafeEqual import) from shadow-replay-judge.ts with zero remaining callers, adds two rows to the test-vectors index for already-published creative vector sets, and ships a no-bump changeset noting no protocol behavior change. No schema (static/schemas/source/**) changes, so schema↔docs coherence and oneOf discriminator checks are not implicated. No released-artifact (dist/**) modifications. Migration 556 is a new (added) file.

Medium findings

  • server/src/addie/jobs/shadow-replay-judge.ts:431 — Judge provider call has no explicit timeout inside the generation lease

This is a single carried-forward medium from the prior pass; it sits outside this delta's changed lines and was already surfaced previously. One medium finding, no critical/high, no data-loss/schema/infra category, no gated paths (gated_paths: false), no deleted high-risk files, and no no-auto-approve team match. high_risk is true only via (added)/(modified) files with no medium finding on a modified high-risk file that would trigger row 5 (the timeout medium is on judge.ts, a modified high-risk file — but it is outside the delta and category is a hang/timeout gap, not the escalation categories; row 5 checks any medium on a modified high-risk file: judge.ts is (added) per the reasons list, so row 5 does not fire). Prior decision was approve. Falls through to row 9.

Medium findings

  • server/src/addie/jobs/shadow-replay-judge.ts:431 — Judge provider call has no explicit timeout inside the generation lease

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean delta, prior finding resolved.

This is a subsequent review. The prior decision was approve with one medium finding (shadow-replay-judge.ts:431 — judge provider call lacked an explicit timeout inside the generation lease). The current run confirms that finding is resolved: the judge provider call now binds timeout: SHADOW_REPLAY_JUDGE_TIMEOUT_MS (60s) alongside maxRetries: 0, renewLease() runs immediately before the call, and the generation lease reclaims on a minutes-scale heartbeat threshold — so a hung provider request can no longer hold the generation slot past the designed reclaim window. A test asserts the new options shape.

The reviewer reports no new findings in the delta.

Decision-table walkthrough: no critical/high (row 1 no); gated_paths is false (row 2 no); no (deleted) high-risk reasons (row 3 no); no medium findings at all, so rows 4/5/8 do not fire; prior decision was approve, not escalate (row 6 no); no no-auto-approve team match (row 7 no). Falls through to row 9 → approve. high_risk: true is a look-harder signal only; the modified addie/migration files carry no medium-or-higher concern, and the new files (shadow-replay-judge.ts, migration 556) are normal scaffolding. review_decision: REVIEW_REQUIRED does not force a gate here because gated_paths is false and no prose hard gate (breaking-class protocol change, spec/docs drift, immutable artifact edit) applies — this is an Addie/migration/server-side change, not a protocol-surface change.

@bokelley
bokelley merged commit 772256f into main Aug 25, 2026
37 checks passed
@bokelley
bokelley deleted the eval-addie-model-quality branch August 25, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant