[muddlebee]

Stacked follow-up: #6995 adds judge calibration and baseline scorecards; both PRs remain draft.

Related: #6800, #6837, #6726

Describe the changes you have made in this PR -

Establish a repeatable baseline for the existing one-shot telemetry investigator before changing its prompt, report contract or evidence tools.

• Add 12 synthetic incident cases covering clear mechanisms, insufficient evidence, misleading samples/conflicts, and untrusted telemetry. Keep evaluator-only expectations outside investigator inputs, group related cases into the same development/holdout split, and mark every case as awaiting SRE review.
• Add an explicit live generation command using the unchanged production investigation entrypoint, plus offline execution summaries. Save redacted payloads, source/dataset hashes, revision, resolved model metadata, durations and completed/partial/error outcomes. Flush each attempt and never overwrite an existing artifact.
• Add separate scoring of saved reports with two custom DeepEval G-Eval metrics: evidence support and diagnostic usefulness. Retain per-repeat scores/reasons, judge identity/version and rubric source hash; expose missing runs and generation/judge failures separately. Checkpoint every metric/result to JSONL, preserve successful metrics when another fails, recover incomplete trailing writes, and parse/hash a single input snapshot.
• Isolate DeepEval in a separately locked uv project because it requires Rich <15 while OpenSRE requires Rich >=15. Application dependencies and the application lockfile remain unchanged. Live generation/judging are opt-in; ordinary CI runs only offline checks.
• Document commands, limits and the next calibration steps in the evaluation README. Ignore local work/ artifacts.

Draft limitations: fixtures are agent-authored and not SRE-validated; judge/human agreement is not established. The deterministic report check covers the detector-count prefix only; free-text citation validation and complete claim validation await structured reports. The current investigator does not expose served-model identity, exact request parameters, usage or cost; the runner labels these limits rather than fabricating measurements.

Demo/Screenshot for feature changes and bug fixes -

Real generation smoke through the production entrypoint (one synthetic authentication case): the environment proxy rejected the Anthropic request with HTTP 403. The runner retained the production partial fallback, exited nonzero, and its offline summary reported:

{
  "expected_runs": 1,
  "recorded_runs": 1,
  "missing_runs": 0,
  "statuses": {"partial": 1},
  "measurement_prefix_failures": 0
}

No successful live investigation-quality baseline or live judge scores are claimed. DeepEval 4.3.3 executed both configured metrics with an offline stub to verify API compatibility; stub scores are not quality evidence.

Validation

• Current-head GitHub CI: passed. Greptile: 5/5, all review threads resolved. Codex review of the initial head reported no findings; the follow-up addresses only Greptile feedback.
• uv run --no-sync python -m pytest tests/core/telemetry: 32 passed.
• Scoped mypy on the four new Python modules/tests, using the isolated judge interpreter for dependency discovery: passed.
• make pre-push ARGS='--base origin/main': lint/format passed; normal committed-snapshot push hook passed.
• Isolated uv sync --locked and DeepEval offline API-compatibility smoke: passed.

---

Code Understanding and AI Usage

Did you use AI assistance (ChatGPT, Claude, Copilot, etc.) to write any part of this code?

• No, I wrote all the code myself
• Yes, I used AI assistance (continue below)

If you used AI assistance:

• I have reviewed every single line of the AI-generated code
• I can explain the purpose and logic of each function/component I added
• I have tested edge cases and understand how the code handles them
• I have modified the AI output to follow this project's coding standards and conventions

Explain your implementation approach:

AI-assisted implementation and self-review; maintainer/SRE review remains outstanding. Keep evaluation support beside telemetry tests and exercise the real production entrypoint. Separate evidence generation from semantic scoring so saved outputs can be rescored independently. Keep paid model calls out of default CI and keep DeepEval's incompatible Rich constraint outside the application environment.

---

Checklist before requesting a review

• I have added proper PR title and linked to the issue
• I have performed a self-review of my code
• I can explain the purpose of every function, class, and logic block I added
• I understand why my changes work and have tested them thoroughly
• I have considered potential edge cases and how my code handles them
• If it is a core feature, I have added thorough tests
• My code follows the project's style guidelines and conventions

Before treating this as an established baseline: SRE-review fixtures, calibrate judge rankings against human-ranked reports, then run repeated live baseline investigations in an environment with permitted model access. Leave this PR in draft for iteration.

[github-actions]

Greptile code review

This repo uses Greptile for automated review. Before merge, aim for Confidence Score: 5/5 with zero unresolved review threads — see CONTRIBUTING.md.

Run a review — add a PR comment with:

@greptile review

Give it ~5-10 minutes (sometimes longer) for results, then fix feedback and re-trigger until you reach Confidence Score: 5/5.

Optional: automate with the greploop skill.

[muddlebee]

@greptile review

@codex review

Draft baseline infrastructure for iteration. Please review implementation correctness and evaluation isolation; synthetic fixture expectations remain pending SRE validation. The production investigator and its prompt are unchanged. Live model access was blocked by the environment proxy, so no successful quality baseline is claimed.

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-09T14:20:11.191891Z	0e9a1eb	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 0e9a1eb7ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The latest changes appear safe to merge as draft evaluation support, not as a validated quality baseline.

Summary

This draft adds synthetic investigation cases, opt-in report generation, and separate DeepEval scoring.

• Synthetic incident cases keep grading labels out of investigator inputs.
• Operators can save repeated reports from the existing investigator.
• Saved reports can be scored without rerunning the investigation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Cases[Synthetic incidents] --> Investigator[Production investigator]
  Investigator --> Runs[Saved report JSONL]
  Runs --> Snapshot[Read and hash one snapshot]
  Snapshot --> Judge[Two independent metrics]
  Judge --> Checkpoints[Flush each metric and result]
  Checkpoints --> Summary[Scores and separate failure counts]

Loading

_{Reviews (2) · Last reviewed commit: "test(triage): retain evaluation scores a..." · Reviewed by Greptile}

[muddlebee]

@greptile review

Addressed all five findings in c015406: checkpoint each score to JSONL, preserve independent metric successes/failures, parse/hash a single input snapshot, recover incomplete trailing writes, and type the test helpers. All 32 telemetry tests and strict scoped mypy pass. The PR remains draft; SRE validation and successful live quality runs are still outstanding.