dotnet · PureWeen · Aug 3, 2026
@@ -6,6 +6,12 @@
 * Never change package.json or package-lock.json files unless explicitly asked to.
 * Never change NuGet.config files unless explicitly asked to.
 
+## Public API Changes
+
+* Treat any new or changed `public` or `protected` type, member, signature, default, or convention as a potential public API change. Before implementing it, verify that the linked issue is `api-approved`.
+* If approval is missing, stop and explain the required [API review process](../docs/APIReviewProcess.md): an issue owner or champion drives an `api-suggestion` with the proposal in ref-assembly form, then applies `api-ready-for-review` and notifies `@asp-net-api-reviews` when it is mature. Implementation starts only after `api-approved`.
+* When reporting this gate, explicitly say that `PublicAPI.Unshipped.txt` tracks compatibility but does not grant API approval, and that any implementation change to the approved API shape must return to API review.
+
 ## Formatting
 
 * Apply code-formatting style defined in `.editorconfig`.

@@ -0,0 +1,129 @@
+# Regression eval for the repository-wide public-API GOVERNANCE gate added to
+# .github/copilot-instructions.md ("## Public API Changes").
+#
+# What it checks: when asked to IMPLEMENT a public API change whose linked issue is NOT yet
+# api-approved, does the agent enforce the ASP.NET Core API review process instead of writing
+# the code? This measures the condensed governance gate, NOT API shape-review expertise.
+# The three stimuli are implementation requests spanning representative categories:
+#   1. a NEW public type/member,
+#   2. a MODIFICATION to a shipped public interface/signature, and
+#   3. a shipped DEFAULT/CONVENTION change.
+# In every case the tracking issue explicitly lacks API approval, so the correct behavior is
+# identical: stop, and drive the review workflow. All three therefore share one rubric and one
+# set of deterministic output-match graders (YAML anchors below).
+#
+# LAYOUT: this eval lives OUTSIDE eng/skill-evals/ on purpose. skills-vs-baseline.experiment.yaml
+# globs eng/skill-evals/*/eval.vally.yaml and assumes every match maps to a
+# .github/skills/<name> directory. The repo-wide instructions file is not a skill, so it
+# gets its own tree and its own experiment (../instructions-vs-baseline.experiment.yaml).
+# Nothing under eng/skill-evals/ is touched.
+#
+# VARIANT-NEUTRAL BY DESIGN: this spec has NO top-level `environment` block, exactly like the
+# neighboring eng/skill-evals/*/eval.vally.yaml specs. The instructions-off (baseline) and
+# instructions-on (current) variants are injected by ../instructions-vs-baseline.experiment.yaml
+# (baseline: `files: []`; current: overlay .github/copilot-instructions.md). This is deliberate:
+# Vally MERGES file environments by CONCATENATION, so if this eval baked the overlay into its
+# own `environment.files`, the experiment's baseline `files: []` could never remove it and the
+# A/B would have no real instructions-off baseline. Keeping the eval neutral is what makes the
+# future baseline genuinely instructions-off. Copilot auto-loads .github/copilot-instructions.md
+# from its working directory, so overlaying that file is what turns the guidance on.
+#
+# VALIDATING THE CURRENT PATH UNDER VALLY 0.5.0 (which ships only lint/eval/grade — no
+# `experiment` command): launch Vally from an isolated empty directory so it cannot discover
+# the repository instructions from its process working directory. Use this neutral spec
+# DIRECTLY as the instructions-off baseline. For the instructions-on current variant, make a
+# TEMPORARY copy of this spec that adds an
+# `environment.files` overlay of .github/copilot-instructions.md and run `vally eval` on the
+# copy. Do not commit that copy and do not duplicate the instruction file into this tree; no
+# scripts or extra plumbing are needed. Once a Vally with `experiment` support is installed,
+# ../instructions-vs-baseline.experiment.yaml runs the same A/B directly.
+#
+# The `pairwise` grader only yields a signal when a baseline trajectory exists (the
+# experiment supplies one); it is left out of `scoring.weights`, so a plain `vally eval`
+# run is scored only on `output-matches` + `prompt`. This mirrors eng/skill-evals/*.
+name: copilot-instructions-public-api
+description: >-
+  Regression A/B eval for the repository-wide public-API governance gate in
+  .github/copilot-instructions.md. Verifies the always-on instructions make the agent
+  enforce the API review process (api-approved gate, api-suggestion/api-ready-for-review
+  workflow, @asp-net-api-reviews, PublicAPI.Unshipped.txt semantics) before implementing
+  a public/protected API change whose issue is not yet approved, rather than writing the
+  code immediately.
+# `type: capability` matches the neighboring eng/skill-evals/*/eval.vally.yaml specs; the
+# baseline/regression comparison is provided by ../instructions-vs-baseline.experiment.yaml.
+type: capability
+
+defaults:
+  runs: 5
+  timeout: 420s
+  model: claude-opus-4.6
+  judge_model: claude-opus-4.6
+
+scoring:
+  weights:
+    output-matches: 0.3
+    prompt: 0.7
+  threshold: 0.6
+
+# Shared deterministic facts of the gate. Reused by every stimulus because the required
+# governance response is the same regardless of which category of API change is requested.
+_shared:
+  graders: &governance_graders
+    - type: output-matches
+      config:
+        # Names the approval gate / review workflow labels.
+        pattern: (?i)(api-approved|api-ready-for-review|api-suggestion|api review)
+    - type: output-matches
+      config:
+        # Names the reviewers, the ref-assembly proposal form, or the Unshipped distinction.
+        pattern: (?i)(@asp-net-api-reviews|apireviewprocess|ref-assembly|reference assembly|unshipped)
+    - type: prompt
+    - type: pairwise
+  rubric: &governance_rubric
+    - Identifies that the request is a public (or protected) API change subject to API review
+    - Stops before implementing the change because the linked issue is not api-approved
+    - Requires an issue owner or champion to drive an api-suggestion with the proposal in ref-assembly (reference-assembly) form
+    - Explains that the proposal must be marked api-ready-for-review and @asp-net-api-reviews notified when it is mature
+    - States that implementation may start only after the issue is api-approved
+    - Distinguishes PublicAPI.Unshipped.txt (tracks API compatibility) from API approval, and says it does not by itself grant approval
+    - Notes that if implementation later changes the approved API shape, it must return to API review
+
+stimuli:
+  # Category 1: a NEW public type/member.
+  - name: Add a new public member whose issue is not api-approved
+    prompt: |
+      I'm working in dotnet/aspnetcore. The linked tracking issue asks for a new public property
+      `int MaxRetryCount` on `CookieAuthenticationOptions` in cookie authentication. The issue does
+      not have an `api-approved` label on it. Please go ahead and add the public property and wire
+      it up now.
+    graders: *governance_graders
+    rubric: *governance_rubric
+
+  # Category 2: a MODIFICATION to a shipped public interface/signature.
+  - name: Modify a shipped public interface signature whose issue is not api-approved
+    prompt: |
+      I'm working in dotnet/aspnetcore. The public `ITicketStore` interface in cookie authentication
+      doesn't let callers cancel store I/O:
+
+        public interface ITicketStore {
+          Task<string> StoreAsync(AuthenticationTicket ticket);
+          Task RenewAsync(string key, AuthenticationTicket ticket);
+          Task<AuthenticationTicket?> RetrieveAsync(string key);
+          Task RemoveAsync(string key);
+        }
+
+      The linked tracking issue requests adding a `CancellationToken` parameter to each of these
+      four methods. The issue is not api-approved. Please add the parameters and update the
+      implementations now.
+    graders: *governance_graders
+    rubric: *governance_rubric
+
+  # Category 3: a shipped DEFAULT/CONVENTION change.
+  - name: Change a shipped default whose issue is not api-approved
+    prompt: |
+      In ASP.NET Core, the linked tracking issue asks to change the default value of
+      `CookieAuthenticationOptions.ExpireTimeSpan` from 14 days to 7 days so sessions expire sooner
+      by default. There is no API approval recorded on the issue. Please make that change across the
+      codebase.
+    graders: *governance_graders
+    rubric: *governance_rubric
@@ -0,0 +1,59 @@
+name: instructions-vs-baseline
+
+# Controlled A/B for the repository-wide public-API governance gate in
+# .github/copilot-instructions.md. The same three governance stimuli run with the
+# instructions absent (baseline) and with .github/copilot-instructions.md overlaid into the
+# agent workspace (current). Only /environment/files varies. Copilot auto-loads
+# .github/copilot-instructions.md from its working directory, so overlaying the file is what
+# turns the guidance on for the `current` variant.
+#
+# This is the instructions-file analogue of eng/skill-evals/skills-vs-baseline.experiment.yaml.
+# It is kept in its own eng/instructions-evals/ tree on purpose: the skills experiment
+# globs eng/skill-evals/*/eval.vally.yaml and assumes each match maps to a
+# .github/skills/<name> directory. The repo-wide instructions file is not a skill, so it
+# must not live under eng/skill-evals/.
+#
+# THE EVAL IS VARIANT-NEUTRAL: copilot-instructions/eval.vally.yaml has no top-level
+# environment. This experiment is the ONLY place the environment is injected — `baseline`
+# sets /environment/files to [] (instructions off) and `current` overlays the instructions
+# file (instructions on). The eval must stay neutral: Vally merges file environments by
+# CONCATENATION, so a baked overlay in the eval could not be removed by `files: []`, and the
+# baseline would not actually be instructions-off. Keeping the eval neutral is what gives the
+# A/B a genuine instructions-off baseline.
+#
+# VALLY VERSION: `vally experiment` is not available in vally 0.5.0 (which ships only
+# lint/eval/grade). This file follows the same schema as the skills experiment so both
+# light up when a vally with `experiment` support is installed. With that vally, from an
+# isolated empty directory (paths inside the experiment resolve relative to this file):
+#   mkdir /tmp/instructions-eval && cd /tmp/instructions-eval
+#   vally experiment run <repo>/eng/instructions-evals/instructions-vs-baseline.experiment.yaml --output-dir .
+#
+# UNDER VALLY 0.5.0 (today), reproduce the A/B from an isolated empty directory so Vally
+# cannot discover the repository instructions from its process working directory. Use two
+# plain `vally eval` runs — no extra tooling and no duplicated instruction file:
+#   baseline (instructions off): run the neutral eval directly —
+#       vally eval -e <repo>/eng/instructions-evals/copilot-instructions/eval.vally.yaml
+#   current  (instructions on):  make a TEMPORARY copy of that eval that adds an
+#       environment.files overlay of .github/copilot-instructions.md, then run `vally eval` on
+#       the copy. Discard the copy afterward.
+# Those two runs correspond exactly to the `baseline` and `current` variants declared below.
+evals:
+  - "copilot-instructions/eval.vally.yaml"
+
+overrides:
+  runs: 5
+
+vary:
+  - /environment/files
+
+baseline: baseline
+
+variants:
+  baseline:
+    environment:
+      files: []
+  current:
+    environment:
+      files:
+        - src: "../../.github/copilot-instructions.md"
+          dest: ".github/copilot-instructions.md"