-
Notifications
You must be signed in to change notification settings - Fork 250
feat(skills): add change-risk model reference for code-review depth selection #2550
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 6 commits
e67e776
ba1b8ab
3c10cfc
c9e819f
8f57304
24d884d
3fad5fb
0fa46b0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| --- | ||
| title: Change-Risk Model | ||
| description: Signal taxonomy and scoring rubric for deterministic, evidence-based change-risk profiling. | ||
| ms.date: 2026-07-29 | ||
| --- | ||
|
|
||
| ## The Change-Risk Profile | ||
|
|
||
| The model produces an advisory **Change-Risk Profile**: a vector of named, evidenced signals, never a single opaque score. To build the profile, evaluate each signal below as High, Medium, or Low based on git history, then combine them into a named-signal profile citing the specific evidence (e.g., "High Diffusion: 12 files across 4 subsystems"). The profile informs the human-scoping step, drives depth-tier selection, and ranks hotspot candidates using deterministic, git-computable signals. | ||
|
|
||
|
|
||
| ## Risk factors | ||
|
|
||
| Evaluate every change across four distinct factors. All signals are computable deterministically using `git log` and `git diff`. | ||
|
|
||
| ### Likelihood | ||
| How likely the change introduces a defect. Measure using: | ||
|
|
||
| * `Size` — lines added or modified. | ||
| * `Diffusion` — number of files, directories, and subsystems touched. | ||
| * `Entropy` — scatter of the change across the codebase. | ||
| * `Hotspot Overlap` — intersection with trailing 90-day high-churn or high-complexity files. | ||
| * `Missed Co-change` — touching one file of a historically coupled cluster while missing its siblings. | ||
|
|
||
| ### Severity | ||
| The blast radius if the change fails. Measure using: | ||
|
|
||
| * `Path Criticality` — classification of touched paths per [Severity Taxonomy](severity-taxonomy.md). | ||
| * *Note: Dependency fan-in and call-graph analysis are deferred to v2 to maintain language-agnostic determinism.* | ||
|
|
||
| ### Detectability | ||
| Whether a defect would be caught before impact. Measure using: | ||
|
|
||
| * `Test Presence` — test files included in the diff. | ||
| * `Coverage` — touched-file test coverage (where CI data is available). | ||
|
|
||
| ### Recoverability | ||
| How quickly the change can be backed out. Measure using: | ||
|
|
||
| * `Reversibility Markers` — presence of feature flags. | ||
| * `Hard-to-Rollback` — schema migrations, database changes, or irreversible config changes versus standard code. | ||
|
|
||
| ## Agentic-era modifiers | ||
|
|
||
| Agent-authored code requires specific adjustments to classic risk models. Apply the following modifiers to the profile: | ||
|
|
||
| * `Provenance-Aware Weighting` — detect agent authorship (e.g., `Co-authored-by:` trailers). Neutralize traditional "author experience" signals, as agents have perfect recall of immediate context but zero memory of historical design intent. | ||
| * `Comprehension Ratio` — evaluate the semantic weight of the delta. A 4,000-line mechanical rename scores lower on Likelihood than 40 lines changing core retry semantics. | ||
| * `Amplification Ratio` — compare the RPI plan artefact delta to the actual code delta. Flag high-risk mismatches where a small plan produces an enormous code delta, or a large plan produces a suspiciously small diff. | ||
|
|
||
| ## Cold starts and confidence | ||
|
|
||
| The model relies on git history. For repositories with shallow history, signal confidence degrades. | ||
|
|
||
| * If trailing 90-day history is insufficient to calculate Hotspot Overlap or Missed Co-change, explicitly state: *"Confidence: Low (shallow history). Relying on Size and Diffusion signals."* | ||
| * Widen confidence intervals and default to standard depth tiers unless Size or Diffusion signals are extreme. Never artificially inflate a risk rating due to missing data. | ||
|
|
||
| ## Gaming and Goodhart's Law | ||
|
|
||
| Agents aware of the scoring model may attempt to game it (e.g., pathologically splitting changes to lower Diffusion scores). | ||
|
|
||
| * The profile is advisory input to a human-confirmed scoping step, not a gate. | ||
| * Batch-splitting that genuinely reduces per-change blast radius is acceptable engineering behavior. | ||
| * The human-scoping protocol evaluates the *intent* of the change, serving as the ultimate safeguard against metric manipulation. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,24 +1,24 @@ | ||
| --- | ||
| title: Code Review Depth Tiers | ||
| description: Basic, standard, and comprehensive review rigor dials for code review perspectives. | ||
| ms.date: 2026-06-18 | ||
| ms.date: 2026-07-31 | ||
| --- | ||
|
|
||
| ## Tier model | ||
|
|
||
| Review depth is a verification-rigor dial, not a lane-selection mechanism. The selected perspectives determine which review lanes run; the selected depth tier determines how deeply each lane verifies the confirmed change scope. | ||
| Review depth is a verification-rigor dial, not a lane-selection mechanism. The selected perspectives determine which review lanes run; the selected depth tier determines how deeply each lane verifies the confirmed change scope. Depth-tier recommendations are driven by the evidence provided in the **Change-Risk Profile** (see [Change-Risk Model](change-risk-model.md)), mapping deterministic, git-computable signals to verification rigor rather than relying on heuristic "gut feel". | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: the |
||
|
|
||
| ## Tier 1 — Basic | ||
|
|
||
| Use Tier 1 when the change is small, low-risk, or time-sensitive. Focus on: | ||
| Use Tier 1 when the Change-Risk Profile indicates low risk. Evidence includes low **Likelihood** (small size, low diffusion, mechanical changes), high **Detectability** (tests present), or high **Recoverability** (safely behind feature flags). Focus on: | ||
|
|
||
| * the primary diff surface, | ||
| * obvious correctness and safety issues, | ||
| * and a quick pass over the main changed files. | ||
|
|
||
| ## Tier 2 — Standard | ||
|
|
||
| Use Tier 2 as the default depth for most reviews. Focus on: | ||
| Use Tier 2 as the default depth for most reviews, or when the Change-Risk Profile indicates moderate risk. Evidence includes moderate **Likelihood** (standard feature work, contained diffusion) with adequate **Detectability** (tests present) and standard **Recoverability** (no hard-to-rollback schema migrations). Focus on: | ||
|
|
||
| * the full changed-file surface, | ||
| * the confirmed hotspot list and adjacent logic, | ||
|
|
@@ -27,7 +27,7 @@ Use Tier 2 as the default depth for most reviews. Focus on: | |
|
|
||
| ## Tier 3 — Comprehensive | ||
|
|
||
| Use Tier 3 for high-risk, high-impact, or ambiguous changes. Focus on: | ||
| Use Tier 3 when the Change-Risk Profile indicates high risk, high impact, or ambiguity. Evidence includes high **Likelihood** and **Severity** (touches trailing 90-day hotspots, high diffusion across subsystems, missed co-changes in coupled clusters, or critical paths per [Severity Taxonomy](severity-taxonomy.md)), low **Detectability** (missing tests for critical logic), or high agentic **Amplification Ratio** (plan-to-code mismatch). Focus on: | ||
|
|
||
| * a deep re-check of the confirmed hotspots and related call paths, | ||
| * broader dependency and regression analysis, | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Heuristics" in step 2 contradicts the "deterministic" framing established in the change-risk-model.md document that this step directly cites. The model opens with: "All signals are computable deterministically using
git logandgit diff." Please replace "heuristics" with "signals":