epic(addie): improve reliability, evaluation integrity, and model portability

[bokelley]

Why

Recent production-thread review found that Addie's visible quality problems are dominated by system behavior rather than a clear Sonnet 5 regression: provider-credit exhaustion, partial/truncated responses, proactive messages attached to unrelated conversations, stale temporal claims, and tool/UI failures.

The same review found an architectural constraint behind both reliability and model portability:

• Addie exposes roughly 169 named tools across 15 routed tool sets.
• 19 tools are always available; knowledge turns can receive 33 tools and admin turns can receive roughly 88.
• The stable system prompt is roughly 200,000 characters, including a 43,000-character prose tool reference that overlaps the concrete tool schemas.
• One Haiku router decision currently blends response policy, capability selection, confidence, and model-depth selection.
• The primary execution loop is coupled directly to Anthropic message, tool-use, web-search, caching, and streaming shapes.

This makes the router carry too much architectural responsibility and makes other providers difficult to evaluate honestly.

Goal

Make Addie reliable under provider and pipeline failures, trustworthy to evaluate, portable across model providers, and cheaper to operate—while preserving one coherent Addie persona and server-side authorization/safety boundaries.

Evidence baseline

• Sonnet 5 production flags were essentially flat versus Sonnet 4.6 in the reviewed window; there is not enough evidence to roll back the model.
• The current shadow dataset is selected from flagged/corrected conversations. Of 31 completed recent cases, 29 were classified as knowledge gaps. This is a useful bug inbox, not a population quality rate.
• Suppressed shadow cases currently allow Haiku to generate a hypothetical no-tools response and then judge it. Corrected cases contain the real production response, but Haiku is still the judge. Generator/judge model metadata is not persisted.
• In an exact router-prompt screening run (24 calls per model), Haiku 4.5 and GPT-5.6 Luna both scored 21/24 under the strict fixture and produced valid JSON 24/24. Luna was about 83% cheaper and 21% faster at median latency.
• A full-prompt Gemini 3.7 Flash screen produced useful answers but had two malformed tool-call failures caused by Anthropic-specific orchestration. This is evidence for an adapter, not evidence that Gemini is unsuitable.

No production transcripts or personal data should be copied into this epic or its children. Evaluation fixtures must be redacted/versioned traces or synthetic equivalents.

Workstreams

0. Reliability and failure containment

• feat(addie): graceful degradation when Anthropic 5xx's mid-stream #4797
• Addie: honor Anthropic Retry-After and surface bounded degradation status #6469
• Addie: define and implement sibling-model fallback policy #6470
• fix(addie): bind proactive account-link messages to their originating thread #6847
• fix(addie): inject authoritative temporal context and date regression tests #6848
• feat(addie): normalize tool results for reliable Slack and web display #6849

1. Evaluation integrity

• fix(addie): make shadow evaluation attributable and non-self-judging #6843

2. Provider-neutral orchestration

• refactor(addie): introduce provider-neutral model and tool orchestration #6844

3. Smaller capability surface and prompt

• refactor(addie): reduce the tool surface and modularize the prompt #6845

4. Cross-provider evaluation and rollout

• test(addie): add cross-provider replay gates and staged model canaries #6846

Architectural constraints

• Keep surface response policy separate from capability selection and model selection.
• Use canonical JSON Schema tool definitions and normalized text/tool/refusal/finish/usage events.
• Keep authorization, confirmation, idempotency, and mutation safety in server-side orchestration.
• Prefer bounded typed domain tools over a universal call_tool(name, args) escape hatch.
• Expose only relevant instructions and tool schemas for a turn; remove the global prose tool catalog as domains migrate.
• Never perform automatic provider/model fallback after an irreversible mutation unless the orchestration layer can prove the turn is safe to replay.
• Treat empty, malformed, truncated, and partial-stream responses as explicit measured failures.

Target shape

deterministic surface policy
  -> small capability planner
  -> 8-12 typed domain tools
  -> provider-neutral execution loop
  -> Anthropic | OpenAI | Gemini | future adapters

Definition of done

• Provider outages and pipeline failures produce bounded, observable recovery behavior instead of silent, partial, or misleading replies.
• Shadow results identify source answer, generator, judge, provider, prompt/config version, tool snapshot/results, and evaluation source.
• Suppressed and corrected-answer evaluations are reported separately and cannot self-judge by default.
• A fixed, versioned, privacy-safe trace suite evaluates answer quality, routing, tool selection, mutation safety, latency, and cost.
• Addie can run the same canonical orchestration against at least Anthropic and one non-Anthropic provider.
• Admin and knowledge turns no longer attach dozens of unrelated tools or a global prose catalog.
• Provider/model changes require passing explicit eval gates and a staged canary with rollback criteria.

Rollout order

1. Reliability fixes and evaluation provenance.
2. Shadow-evaluation semantic repair.
3. Provider-neutral runtime with no intended behavior change.
4. Domain-by-domain tool and prompt consolidation, beginning with admin.
5. Luna router shadow/canary, then Gemini full-response shadow/canary.

This is application/runtime work and does not change the published AdCP protocol surface.