[muddlebee]

Stacked on #6994. Base branch: codex/triage-investigation-eval-baseline, so this PR shows only the calibration/reporting increment. Keep draft; merge the parent first, then rebase onto main and retarget after it lands.

Related: #6837, #6800, #6726.

Describe the changes you have made in this PR -

Make the evaluation baseline reviewable by an SRE and make saved results easier to assess before changing the investigator.

• Add nine authored calibration reports across three development cases, contrasting supported findings, unnecessary abstention, and unsupported causal narratives. Every proposed assessment remains pending SRE review.
• Add offline review-pack preparation: neutral stable candidate IDs, seeded shuffled ordering, sanitized reports and payloads, and a human worksheet with pending/null ratings. Author notes are omitted from human and judge inputs. Existing DeepEval scoring handles the pack without seeing human ratings.
• Compare attributed human ratings with judge preferences within each case/metric. Require accepted case expectations and completed ratings; retain ties, missing judgments and disagreements. No eligible comparisons means null agreement, not perfect agreement.
• Add offline failure-class scorecards for saved investigator runs. Bind score artifacts to the exact generation snapshot, validate checkpoint/result consistency, retain interrupted metrics, and expose generation/judge failures and missing attempts. Count generation contract failures independently of judging; report expected/missing counts for every recoverable class and explicitly list unclassified selected cases. Reject authored calibration packs as investigator baseline results.
• Document the commands and proposed rating anchors in CALIBRATION.md. No product investigator, prompt, model dependencies, or paid CI calls change.

What remains pending: actual SRE case review/ratings, human/judge agreement, and successful repeated live investigator/judge runs. This PR adds the workflow and scorecard; it does not claim a calibrated judge, an established baseline v0, or production accuracy. The prior model smoke was blocked by the environment proxy. All source fixture review statuses remain unchanged.

Demo/Screenshot for feature changes and bug fixes -

Real offline pack preparation and comparison with an explicitly stubbed judge:

Authored reports: 9; SRE-reviewed ratings: 0; agreement: null.
Offline CLI smoke uses stub judge scores; no live calibration or investigator quality claim.

The scorecard CLI also consumed the parent's retained partial-generation smoke plus offline-stub judge artifact and preserved the partial generation status.

Validation

• Greptile 5/5 on b941db9d4, zero unresolved threads. Codex follow-up on 9455b4846 found no further issues; the subsequent change addresses only Greptile coverage feedback.
• GitHub CI is not scheduled for this stacked base: the Linux/Windows workflows filter PR bases to main. Keep this PR stacked; full CI must run after retargeting to main once test(triage): establish investigation evaluation baseline #6994 lands. Local passing checks below are not represented as remote CI.
• uv run --no-sync python -m pytest tests/core/telemetry: 48 passed, including 16 new calibration/reporting cases.
• Strict scoped mypy (--check-untyped-defs --disallow-untyped-defs) on the three new Python modules/tests: passed.
• make pre-push ARGS='--base codex/triage-investigation-eval-baseline': lint/format passed.
• Real prepare/compare and scorecard CLI smoke using offline judge stubs: passed; no live quality scores.

---

Code Understanding and AI Usage

Did you use AI assistance (ChatGPT, Claude, Copilot, etc.) to write any part of this code?

• No, I wrote all the code myself
• Yes, I used AI assistance (continue below)

If you used AI assistance:

• I have reviewed every single line of the AI-generated code
• I can explain the purpose and logic of each function/component I added
• I have tested edge cases and understand how the code handles them
• I have modified the AI output to follow this project's coding standards and conventions

Explain your implementation approach:

AI-assisted implementation and self-review; maintainer/SRE review remains outstanding. Reuse the parent's generation/scoring formats and isolated judge. Keep authored calibration examples explicitly distinct from investigator outputs. Compare ordinal preferences within a case instead of treating human and judge score scales as interchangeable. Human judgments are external inputs with attribution and snapshot binding; the implementation does not supply or fabricate them.

---

Checklist before requesting a review

• I have added proper PR title and linked to the issue
• I have performed a self-review of my code
• I can explain the purpose of every function, class, and logic block I added
• I understand why my changes work and have tested them thoroughly
• I have considered potential edge cases and how my code handles them
• If it is a core feature, I have added thorough tests
• My code follows the project's style guidelines and conventions

This is a stacked draft for iteration, not a request to merge either PR.

[github-actions]

Greptile code review

This repo uses Greptile for automated review. Before merge, aim for Confidence Score: 5/5 with zero unresolved review threads — see CONTRIBUTING.md.

Run a review — add a PR comment with:

@greptile review

Give it ~5-10 minutes (sometimes longer) for results, then fix feedback and re-trigger until you reach Confidence Score: 5/5.

Optional: automate with the greploop skill.

[muddlebee]

@greptile review

@codex review

Stacked draft on #6994; please review only the calibration/reporting increment against its base branch. Focus on provenance, missing/failing judgment accounting, human-rating isolation and scorecard correctness. All proposed SRE judgments and successful live baseline results remain explicitly pending.

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-09T15:23:35.951751Z	9455b48	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a0a8f64da3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The reviewed changes appear safe to merge, subject to the stated stacked-PR process.

Summary

Adds authored judge-calibration packs, human/judge comparisons, and offline scorecards for saved investigator runs.

• SREs can compare authored reports with judge preferences.
• Saved investigator runs produce failure-class scorecards.

_{Reviews (2) · Last reviewed commit: "test(triage): expose missing generation ..." · Reviewed by Greptile}

[muddlebee]

@codex review

Fixed the generation-contract accounting finding in 9455b48 with an interrupted-judging regression. Prefix failures now count all recorded generation attempts, including attempts never reached by the judge. The updated 14 calibration/reporting tests and strict scoped mypy pass; the stacked base and pending human/live validation remain unchanged.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 9455b48469

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[muddlebee]

@greptile review

Addressed missing-class coverage in b941db9. Class scorecards show planned/recorded/missing counts, including zero-run classes from a verified case dataset or an explicit unclassified selected-case bucket when the original dataset is unavailable. Empty classes have no duration/semantic score. Both paths are covered by regressions; 16 calibration/reporting tests and strict scoped mypy pass.