fix(addie): make shadow evaluation attributable and non-self-judging

[bokelley]

Parent epic: #6842

Problem

Addie's current shadow-evaluation records combine two semantically different sources:

1. Suppressed opportunity: Addie did not answer because a human answered first. A fast model generates a hypothetical response without production tool execution, then the same model may judge that response.
2. Corrected answer: Addie's real production response is followed by a substantive human reply. A fast model judges the production response against that reply.

These are useful signals, but they are not comparable quality samples. The suppressed path is especially biased for questions that require retrieval or account/admin tools. Records also do not persist enough generator/judge/provider/config provenance to support later stratification.

In a recent selected sample, 29 of 31 completed flagged evaluations were labeled knowledge gaps (23/24 corrected-answer captures and 6/7 suppressed captures). That is a bug-discovery queue, not a population failure rate.

Scope

• Give the two sources explicit evaluation types and separate reporting.
• Persist source-answer, generator, judge, provider, prompt/config version, tool-set/schema snapshot, and tool-result provenance.
• Prevent a model from judging its own generated answer by default.
• Replay suppressed opportunities through production-equivalent orchestration with safe/read-only tools or recorded tool fixtures. Do not merely show tool descriptions to a model that cannot call them.
• Grade deterministic failures before using an LLM judge: blank output, malformed structured output, wrong tool/mutation class, truncation, and response-shape violations.
• Keep raw production transcripts and personal data out of issues and versioned fixtures.

Acceptance criteria

• Suppressed and corrected-answer evaluations have distinct stored/reportable types.
• Every result records source-answer model when applicable, generator model/provider, judge model/provider, prompt/config version, and timestamp.
• Tool sets alone are not treated as proof of tool availability; replay records the exposed schemas and tool results or the fixture identifier used.
• The default judge is independent of the generator/source model; self-judging requires an explicit experimental flag and is excluded from headline metrics.
• Suppressed evaluation uses production-equivalent orchestration in dry-run/read-only mode, with mutation attempts blocked and recorded.
• Corrected-answer evaluation grades the actual production answer rather than regenerating a substitute.
• Blank, malformed, truncated, and orchestration failures count as failures rather than disappearing from the denominator.
• Reports show source-specific sample sizes and selection-bias warnings.
• Migration/backfill behavior for existing rows is explicit and tested.

Delivery slices

1. Provenance schema + writes + source-separated reporting — fix(addie): make shadow evaluations attributable #6852
2. Independent judge and deterministic grader pipeline — fix(addie): make shadow evaluations attributable #6852
3. Safe production-orchestration replay for suppressed opportunities.

This is Addie application work; no protocol changeset is required.

[bokelley]

Triage

Classification: Bug
Bucket(s): addie
Status: deferred

Blocked-on: #6842 — resurfaces on close.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_014qsey557KrsFg9Zz8XLMqq

---

Generated by Claude Code

[bokelley]

Merged the evaluation-integrity foundation in #6852 (merge commit d0dbd2f).

Delivered: attributable source/model/tool provenance; independent judging by default; corrected-answer capture and keyset backfill; fail-closed invalid/legacy/self-judged handling; explicit skipped disposition for over-bound evidence; transcript-free public issue filing with retry-safe backoff; and denominator/provenance reporting tests. All CI checks passed, both review threads were resolved, and expert review signed off.

This issue remains open for the next slice: safe, read-only production-equivalent orchestration replay and representative fixtures.

[bokelley]

Milestone: PR #6859 merged green as the fail-closed production replay foundation. It shares Addie production orchestration, classifies replay-safe tools, records HMAC-only prompt/schema/input/result provenance, and strictly excludes incomplete evidence from judging/headline metrics. Review-driven hardening now prevents any model call unless preflight evidence—including an independently verified trace—is complete, so the default scheduled path spends zero generation tokens. All 29 CI checks passed; both review threads were addressed and resolved. Next slice: implement a restricted, attributable trace/fixture source so complete replays can run safely.

[bokelley]

PR #6863 merged as 78f1f5a2b4, completing the fail-closed signed replay-capture foundation for delivery slice 3.

Delivered:

• immutable, expiring HMAC-signed replay traces bound to the source thread/message
• exact no-provider invocation preparation and docs-corpus/tool-schema fingerprints
• server-side authorization before private source hydration
• removal/redaction of copied private replay inputs from mutable thread JSON
• fail-closed provider-web/SI checks, retry-safe queue accounting, and migration fixtures
• headline/public-gap admission remains closed unless the signed source is verified

The automated review's sole observability finding was fixed and its thread resolved. Final CI passed all unit, integration, migration, Docker/docs-index, CodeQL, and canonical gates.

Generation is intentionally still disabled (signed_trace_activation_pending): current production opportunities expose provider web search, while safe replay disables it, so those answers would not be production-equivalent evidence. The remaining slice is a production docs-only cohort with matching provider-tool state, followed by the signed executor and output comparison.