Skip to content

Calibrate and risk-tier independent artifact review #490

Description

@jeremymanning

Origin

Deferred from the MVP proposed as a fresh realization of #485.

Goal

Evolve the MVP's bounded plan/final-output review into a calibrated, risk-tiered review service.

Scope

  • Per reviewer model/prompt/version precision, recall, false-positive rate, abstention, and cost metrics.
  • Seeded-defect and adjudicated production datasets for calibration.
  • Risk tiers: deterministic checks, sampling/differential review, mandatory independent review, and human adjudication.
  • Model-family versus evidence-channel independence experiments.
  • Dispute handling and accepted-risk policy.
  • Review of intermediate artifacts where measured benefit justifies cost.

Acceptance criteria

  • Reviewers are evaluated against a versioned benchmark and production adjudications.
  • Policy selects review depth from consequence/risk and measured reviewer quality.
  • Review-loop non-convergence and cost stay within declared bounds.
  • A pass reports residual risk; it never claims that an artifact is simply "clean."

Metadata

Metadata

Assignees

No one assigned

    Labels

    architectureRelated to toolbox design/architectureenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions