From e67e77686f77423c1713bb754b72c97e2a32bc8d Mon Sep 17 00:00:00 2001 From: pratikwayase Date: Wed, 29 Jul 2026 15:30:15 +0530 Subject: [PATCH 1/3] feat(skills): add change-risk model reference for code-review depth selection --- .../references/change-risk-model.md | 64 +++++++++++++++++++ .../references/context-bootstrap.md | 16 +++-- .../code-review/references/depth-tiers.md | 8 +-- .../references/severity-taxonomy.md | 2 + 4 files changed, 81 insertions(+), 9 deletions(-) create mode 100644 .github/skills/coding-standards/code-review/references/change-risk-model.md diff --git a/.github/skills/coding-standards/code-review/references/change-risk-model.md b/.github/skills/coding-standards/code-review/references/change-risk-model.md new file mode 100644 index 000000000..960c1cf6b --- /dev/null +++ b/.github/skills/coding-standards/code-review/references/change-risk-model.md @@ -0,0 +1,64 @@ +--- +title: Change-Risk Model +description: Signal taxonomy and scoring rubric for deterministic, evidence-based change-risk profiling. +ms.date: 2026-07-29 +--- + +## The Change-Risk Profile + +The model produces an advisory **Change-Risk Profile**: a vector of named, evidenced signals, never a single opaque score. To build the profile, evaluate each signal below as High, Medium, or Low based on git history, then combine them into a named-signal profile citing the specific evidence (e.g., "High Diffusion: 12 files across 4 subsystems"). The profile informs the human-scoping step, drives depth-tier selection, and ranks hotspot candidates using deterministic, git-computable signals. + + +## Risk factors + +Evaluate every change across four distinct factors. All signals are computable deterministically using `git log` and `git diff`. + +### Likelihood +How likely the change introduces a defect. Measure using: + +* `Size` — lines added or modified. +* `Diffusion` — number of files, directories, and subsystems touched. +* `Entropy` — scatter of the change across the codebase. +* `Hotspot Overlap` — intersection with trailing 90-day high-churn or high-complexity files. +* `Missed Co-change` — touching one file of a historically coupled cluster while missing its siblings. + +### Severity +The blast radius if the change fails. Measure using: + +* `Path Criticality` — classification of touched paths per [Severity Taxonomy](severity-taxonomy.md). +* *Note: Dependency fan-in and call-graph analysis are deferred to v2 to maintain language-agnostic determinism.* + +### Detectability +Whether a defect would be caught before impact. Measure using: + +* `Test Presence` — test files included in the diff. +* `Coverage` — touched-file test coverage (where CI data is available). + +### Recoverability +How quickly the change can be backed out. Measure using: + +* `Reversibility Markers` — presence of feature flags. +* `Hard-to-Rollback` — schema migrations, database changes, or irreversible config changes versus standard code. + +## Agentic-era modifiers + +Agent-authored code requires specific adjustments to classic risk models. Apply the following modifiers to the profile: + +* `Provenance-Aware Weighting` — detect agent authorship (e.g., `Co-authored-by:` trailers). Neutralize traditional "author experience" signals, as agents have perfect recall of immediate context but zero memory of historical design intent. +* `Comprehension Ratio` — evaluate the semantic weight of the delta. A 4,000-line mechanical rename scores lower on Likelihood than 40 lines changing core retry semantics. +* `Amplification Ratio` — compare the RPI plan artefact delta to the actual code delta. Flag high-risk mismatches where a small plan produces an enormous code delta, or a large plan produces a suspiciously small diff. + +## Cold starts and confidence + +The model relies on git history. For repositories with shallow history, signal confidence degrades. + +* If trailing 90-day history is insufficient to calculate Hotspot Overlap or Missed Co-change, explicitly state: *"Confidence: Low (shallow history). Relying on Size and Diffusion signals."* +* Widen confidence intervals and default to standard depth tiers unless Size or Diffusion signals are extreme. Never artificially inflate a risk rating due to missing data. + +## Gaming and Goodhart's Law + +Agents aware of the scoring model may attempt to game it (e.g., pathologically splitting changes to lower Diffusion scores). + +* The profile is advisory input to a human-confirmed scoping step, not a gate. +* Batch-splitting that genuinely reduces per-change blast radius is acceptable engineering behavior. +* The human-scoping protocol evaluates the *intent* of the change, serving as the ultimate safeguard against metric manipulation. diff --git a/.github/skills/coding-standards/code-review/references/context-bootstrap.md b/.github/skills/coding-standards/code-review/references/context-bootstrap.md index f853f2ae1..0b12b9339 100644 --- a/.github/skills/coding-standards/code-review/references/context-bootstrap.md +++ b/.github/skills/coding-standards/code-review/references/context-bootstrap.md @@ -15,11 +15,12 @@ Start with the orientation floor from [Walkthrough Protocol](walkthrough-protoco ## Tier 0 procedure 1. Compute the diff once from the selected base branch and capture the changed-file surface. -2. Summarize the change in a concise change brief that explains what changed and why it matters. -3. Auto-detect hotspot candidates and specialist concern signals from the diff and file paths in the same pass. Tag the specialist concern classes for security, supply-chain, RAI or AI, accessibility, sustainability or efficiency, and privacy or PII using the signal-to-concern mapping in [Cross-Skill Forks](cross-skill-forks.md). -4. Present the emerging brief and hotspot candidates to the human for confirmation and correction. -5. Invite the human to add or remove hotspots and to mark out-of-scope areas before review lanes dispatch. -6. Persist the confirmed brief, the scoped hotspot list, the tagged specialist concerns, and out-of-scope areas as the review context for later aggregation. +2. Draft a **Change-Risk Profile** alongside the change brief using the deterministic signals defined in [Change-Risk Model](change-risk-model.md). Evaluate Likelihood (size, diffusion, entropy), Severity (path criticality), Detectability (test presence), and Recoverability (rollback markers) using `git log` and `git diff` heuristics. +3. Summarize the change in a concise change brief that explains what changed and why it matters. +4. Auto-detect hotspot candidates and specialist concern signals from the diff and file paths in the same pass. Tag the specialist concern classes for security, supply-chain, RAI or AI, accessibility, sustainability or efficiency, and privacy or PII using the signal-to-concern mapping in [Cross-Skill Forks](cross-skill-forks.md). Rank hotspot candidates using the risk profile's evidence (e.g., trailing churn, missed co-changes). +5. Present the emerging brief, the Change-Risk Profile evidence, and hotspot candidates to the human for confirmation and correction. +6. Invite the human to add or remove hotspots, adjust the risk profile interpretation, and mark out-of-scope areas before review lanes dispatch. +7. Persist the confirmed brief, the risk profile, the scoped hotspot list, the tagged specialist concerns, and out-of-scope areas as the review context for later aggregation. ## Change brief expectations @@ -30,11 +31,16 @@ The change brief should be short and specific. It should explain: * the likely risk areas, * and any notable test or rollout considerations. +## Change-Risk Profile expectations + +The Change-Risk Profile is an advisory, evidence-bearing vector, never a single opaque score. It must cite specific git-computable signals (e.g., "High diffusion across 4 subsystems", "Touches trailing 90-day hotspots", "No test files present in diff"). For repositories with shallow git history, explicitly state the confidence level and rely on baseline signals like Size and Diffusion. The profile informs the human-scoping step and subsequent depth-tier selection but never acts as a hard gate. + ## Human-scoping protocol Do not let the agent decide the entire scope alone. The human should be able to: * confirm or edit the change brief, +* review and adjust the Change-Risk Profile evidence, * add or remove hotspot candidates, * and explicitly mark areas that should not be reviewed in this run. diff --git a/.github/skills/coding-standards/code-review/references/depth-tiers.md b/.github/skills/coding-standards/code-review/references/depth-tiers.md index 690f1ed03..df267fc75 100644 --- a/.github/skills/coding-standards/code-review/references/depth-tiers.md +++ b/.github/skills/coding-standards/code-review/references/depth-tiers.md @@ -6,11 +6,11 @@ ms.date: 2026-06-18 ## Tier model -Review depth is a verification-rigor dial, not a lane-selection mechanism. The selected perspectives determine which review lanes run; the selected depth tier determines how deeply each lane verifies the confirmed change scope. +Review depth is a verification-rigor dial, not a lane-selection mechanism. The selected perspectives determine which review lanes run; the selected depth tier determines how deeply each lane verifies the confirmed change scope. Depth-tier recommendations are driven by the evidence provided in the **Change-Risk Profile** (see [Change-Risk Model](change-risk-model.md)), mapping deterministic, git-computable signals to verification rigor rather than relying on heuristic "gut feel". ## Tier 1 — Basic -Use Tier 1 when the change is small, low-risk, or time-sensitive. Focus on: +Use Tier 1 when the Change-Risk Profile indicates low risk. Evidence includes low **Likelihood** (small size, low diffusion, mechanical changes), high **Detectability** (tests present), or high **Recoverability** (safely behind feature flags). Focus on: * the primary diff surface, * obvious correctness and safety issues, @@ -18,7 +18,7 @@ Use Tier 1 when the change is small, low-risk, or time-sensitive. Focus on: ## Tier 2 — Standard -Use Tier 2 as the default depth for most reviews. Focus on: +Use Tier 2 as the default depth for most reviews, or when the Change-Risk Profile indicates moderate risk. Evidence includes moderate **Likelihood** (standard feature work, contained diffusion) with adequate **Detectability** (tests present) and standard **Recoverability** (no hard-to-rollback schema migrations). Focus on: * the full changed-file surface, * the confirmed hotspot list and adjacent logic, @@ -27,7 +27,7 @@ Use Tier 2 as the default depth for most reviews. Focus on: ## Tier 3 — Comprehensive -Use Tier 3 for high-risk, high-impact, or ambiguous changes. Focus on: +Use Tier 3 when the Change-Risk Profile indicates high risk, high impact, or ambiguity. Evidence includes high **Likelihood** and **Severity** (touches trailing 90-day hotspots, high diffusion across subsystems, missed co-changes in coupled clusters, or critical paths per [Severity Taxonomy](severity-taxonomy.md)), low **Detectability** (missing tests for critical logic), or high agentic **Amplification Ratio** (plan-to-code mismatch). Focus on: * a deep re-check of the confirmed hotspots and related call paths, * broader dependency and regression analysis, diff --git a/.github/skills/coding-standards/code-review/references/severity-taxonomy.md b/.github/skills/coding-standards/code-review/references/severity-taxonomy.md index caabe680d..24fbaec06 100644 --- a/.github/skills/coding-standards/code-review/references/severity-taxonomy.md +++ b/.github/skills/coding-standards/code-review/references/severity-taxonomy.md @@ -29,6 +29,8 @@ Assign file-level risk using the component context: * `Medium` for core business logic, API boundaries, and shared utilities with broad impact. * `Low` for configuration, documentation, cosmetic changes, and isolated helper code. +> **Note:** The path-based classification above serves as the **Severity** factor input for the broader quantitative **Change-Risk Model** (see [Change-Risk Model](change-risk-model.md)). Path criticality is one of four factors — alongside Likelihood, Detectability, and Recoverability — used to build the evidence-based Change-Risk Profile. It is no longer the sole determinant of a change's overall risk. + ## Severity count convention Aggregate findings into `severity_counts` with the counts for `critical`, `high`, `medium`, and `low`. When a finding is not applicable to the chosen perspective, omit it from that perspective-specific report but preserve it in the merged report if it was surfaced by another lane. From c9e819fc85f48a63f54d9aafb7bef6b564a5bf37 Mon Sep 17 00:00:00 2001 From: pratikwayase Date: Fri, 31 Jul 2026 11:56:21 +0530 Subject: [PATCH 2/3] chore: address review feedback for change-risk-model reference --- .github/skills/coding-standards/code-review/SKILL.md | 4 +++- .../code-review/references/context-bootstrap.md | 2 +- .../coding-standards/code-review/references/depth-tiers.md | 2 +- .../code-review/references/severity-taxonomy.md | 2 +- 4 files changed, 6 insertions(+), 4 deletions(-) diff --git a/.github/skills/coding-standards/code-review/SKILL.md b/.github/skills/coding-standards/code-review/SKILL.md index 89788e435..cdeb1709f 100644 --- a/.github/skills/coding-standards/code-review/SKILL.md +++ b/.github/skills/coding-standards/code-review/SKILL.md @@ -6,7 +6,7 @@ user-invocable: true metadata: authors: "microsoft/hve-core" spec_version: "1.0" - last_updated: "2026-06-18" + last_updated: "2026-07-31" --- # Code Review — Skill Entry @@ -30,6 +30,7 @@ Review work should stay anchored in evidence and should avoid premature conclusi 7. [Dispatch Loop](references/dispatch-loop.md) — human-steered dispatch board, manifest schema, and walk-back loop contract. 8. [Emission Modes](references/emission-modes.md) — capability-gated dual-mode emission and persisted emission record. 9. [Cross-Skill Forks](references/cross-skill-forks.md) — specialist review registry and collection-aware gating for follow-up reviews. +10. [Change-Risk Model](references/change-risk-model.md) — deterministic signal taxonomy and scoring rubric for evidence-based change-risk profiling. ## Skill layout @@ -44,3 +45,4 @@ Review work should stay anchored in evidence and should avoid premature conclusi * `dispatch-loop.md` — dispatch board, manifest schema, and walk-back loop. * `emission-modes.md` — native and canonical emission strategies. * `cross-skill-forks.md` — specialist review registry and gating rules. + * `change-risk-model.md` — deterministic signal taxonomy and scoring rubric for evidence-based change-risk profiling. diff --git a/.github/skills/coding-standards/code-review/references/context-bootstrap.md b/.github/skills/coding-standards/code-review/references/context-bootstrap.md index 0b12b9339..d2d184a57 100644 --- a/.github/skills/coding-standards/code-review/references/context-bootstrap.md +++ b/.github/skills/coding-standards/code-review/references/context-bootstrap.md @@ -1,7 +1,7 @@ --- title: Code Review Context Bootstrap description: Tier 0 workflow for establishing the change surface, drafting a change brief, and scoping review hotspots. -ms.date: 2026-06-26 +ms.date: 2026-07-31 --- ## Objective diff --git a/.github/skills/coding-standards/code-review/references/depth-tiers.md b/.github/skills/coding-standards/code-review/references/depth-tiers.md index df267fc75..91b128d49 100644 --- a/.github/skills/coding-standards/code-review/references/depth-tiers.md +++ b/.github/skills/coding-standards/code-review/references/depth-tiers.md @@ -1,7 +1,7 @@ --- title: Code Review Depth Tiers description: Basic, standard, and comprehensive review rigor dials for code review perspectives. -ms.date: 2026-06-18 +ms.date: 2026-07-31 --- ## Tier model diff --git a/.github/skills/coding-standards/code-review/references/severity-taxonomy.md b/.github/skills/coding-standards/code-review/references/severity-taxonomy.md index 24fbaec06..e1f8c4335 100644 --- a/.github/skills/coding-standards/code-review/references/severity-taxonomy.md +++ b/.github/skills/coding-standards/code-review/references/severity-taxonomy.md @@ -1,7 +1,7 @@ --- title: Code Review Severity Taxonomy description: Severity levels, verdict normalization, and risk classification guidance for code review findings. -ms.date: 2026-06-18 +ms.date: 2026-07-31 --- ## Severity levels From 0fa46b08f05fd119e7cc554a4851f433e9f427d7 Mon Sep 17 00:00:00 2001 From: pratikwayase Date: Sun, 2 Aug 2026 11:22:51 +0530 Subject: [PATCH 3/3] fix: address reviewer suggestions and merge main into feat/change-risk-model --- .../code-review/references/context-bootstrap.md | 2 +- .../code-review/references/severity-taxonomy.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/skills/coding-standards/code-review/references/context-bootstrap.md b/.github/skills/coding-standards/code-review/references/context-bootstrap.md index d2d184a57..1208c21eb 100644 --- a/.github/skills/coding-standards/code-review/references/context-bootstrap.md +++ b/.github/skills/coding-standards/code-review/references/context-bootstrap.md @@ -15,7 +15,7 @@ Start with the orientation floor from [Walkthrough Protocol](walkthrough-protoco ## Tier 0 procedure 1. Compute the diff once from the selected base branch and capture the changed-file surface. -2. Draft a **Change-Risk Profile** alongside the change brief using the deterministic signals defined in [Change-Risk Model](change-risk-model.md). Evaluate Likelihood (size, diffusion, entropy), Severity (path criticality), Detectability (test presence), and Recoverability (rollback markers) using `git log` and `git diff` heuristics. +2. Draft a **Change-Risk Profile** alongside the change brief using the deterministic signals defined in [Change-Risk Model](change-risk-model.md). Evaluate Likelihood (size, diffusion, entropy), Severity (path criticality), Detectability (test presence), and Recoverability (rollback markers) using `git log` and `git diff` signals. 3. Summarize the change in a concise change brief that explains what changed and why it matters. 4. Auto-detect hotspot candidates and specialist concern signals from the diff and file paths in the same pass. Tag the specialist concern classes for security, supply-chain, RAI or AI, accessibility, sustainability or efficiency, and privacy or PII using the signal-to-concern mapping in [Cross-Skill Forks](cross-skill-forks.md). Rank hotspot candidates using the risk profile's evidence (e.g., trailing churn, missed co-changes). 5. Present the emerging brief, the Change-Risk Profile evidence, and hotspot candidates to the human for confirmation and correction. diff --git a/.github/skills/coding-standards/code-review/references/severity-taxonomy.md b/.github/skills/coding-standards/code-review/references/severity-taxonomy.md index e1f8c4335..377e628ed 100644 --- a/.github/skills/coding-standards/code-review/references/severity-taxonomy.md +++ b/.github/skills/coding-standards/code-review/references/severity-taxonomy.md @@ -29,7 +29,7 @@ Assign file-level risk using the component context: * `Medium` for core business logic, API boundaries, and shared utilities with broad impact. * `Low` for configuration, documentation, cosmetic changes, and isolated helper code. -> **Note:** The path-based classification above serves as the **Severity** factor input for the broader quantitative **Change-Risk Model** (see [Change-Risk Model](change-risk-model.md)). Path criticality is one of four factors — alongside Likelihood, Detectability, and Recoverability — used to build the evidence-based Change-Risk Profile. It is no longer the sole determinant of a change's overall risk. +> **Note:** The path-based classification above serves as the **Severity** factor input for the broader quantitative **Change-Risk Model** (see [Change-Risk Model](change-risk-model.md)). Path criticality is one of four factors (alongside Likelihood, Detectability, and Recoverability) used to build the evidence-based Change-Risk Profile. Path criticality is one signal within a broader risk profile and does not determine overall change risk on its own. ## Severity count convention