feat(agents): consolidate eval-dataset-creator docs into one guide and close RAI gaps

[Bill Berry (WilliamBerryiii)]

feat(agents): consolidate eval-dataset-creator docs into one guide and close RAI gaps

Description

The Evaluation Dataset Creator agent emitted 5 files per run: two dataset serializations plus three separate Markdown documents. This PR merged those three documents into a single sectioned {agent-name}-eval-guide.md, taking the run to 3 files while leaving the dataset dual-format output untouched. A Responsible AI review of the same agent surfaced six additive gaps in the same two files, so both bodies of work closed here together.

The agent builds a measuring instrument. That shaped the Responsible AI half of this change: a gap here yields false assurance about some other system rather than an obviously wrong answer, so every remediation landed in the NIST AI RMF Measure function and every edit was additive.

Consolidation

• Reduced the ## Output Artifacts tree to one data/evaluation/docs/{agent-name}-eval-guide.md entry, replacing the three separate document filenames.
• Merged the three Phase 7 template blocks into one <eval-guide-template> whose H2 sections are ## Curation Notes, ## Metric Selection, and ## Tool Recommendations. Heading levels were demoted uniformly beneath the new guide H1 (former H1 to H2, H2 to H3, H3 to H4), preserving every subsection, table, and checklist.
• Repointed the cross-reference in the Azure AI Foundry getting-started steps from "selection document" to the Metric Selection section.

Responsible AI remediation

• Routed the Responsible AI interview answer to a deliverable. Question 5 now states that each named risk drives metric selection, and the guide's Metric Selection section carries a ### Responsible AI Risks table mapping each risk to the metric selected to detect it. A risk with no detecting metric is stated explicitly rather than dropped.
• Added the missing #### Responsibility and Safety Metrics table (Fairness, Harmful Content, Groundedness under adversarial framing) with matching entries in the Metric Definitions Reference. The agent previously advertised Azure AI Foundry Risk/Safety metrics in its comparison table while offering no way to select one.
• Gave representativeness a real schema axis. metadata.population_coverage sits as a sibling of distribution, keyed by the populations named in the interview, and Question 6 now asks for distinct user populations rather than job titles alone. The generation rule states that population is its own axis and is never expressed as a difficulty value or a distribution key.
• Recorded provenance. metadata.validation_status (ai-generated, expert-reviewed, or mixed, defaulting to ai-generated) and metadata.generation_method were added, a new Required Protocol item states that the dataset is provisional while the status is ai-generated, and the guide's Domain Expert Review subsection now states that completing its checkboxes warrants updating the status.
• Bounded generated content. Dataset Requirements now requires synthesis rather than reproduction of real records or personal data, and requires safety and negative pairs to record the disallowed request category and the expected refusal rather than the prohibited content itself.

Eval spec

• Renamed three-supporting-docs to consolidated-eval-guide and rewrote its signal, pass criteria, failure modes, and contract reference to assert one guide carrying its three sections. Reverting to three separate files was retained as an explicit failure mode.
• Repointed persona-driven-tool-recommendation and output-path-under-data-evaluation from the retired filenames to the guide and its Tool Recommendations section.
• Added one expectation, rai-metadata-completeness, asserting the three new metadata fields, with failure modes for each missing field, for an out-of-set validation_status, and for expressing population as a sixth difficulty value.

Compatibility

The graded difficulty enum and the distribution keys were deliberately left untouched. dataset-schema-conformance names out-of-set difficulty values as an explicit failure mode, so adding a fairness category, the intuitive fix, would have failed the eval on contact. Its schema assertion also requires its listed fields "at minimum", so the three new metadata keys pass without an eval change. Every pre-existing expectation other than the three named above is byte-unchanged, and datasets already produced under the current contract stay valid.

The new Required Protocol item was appended as item 12 rather than inserted, because interview-driven-flow, dual-format-dataset-output, and phase-transition-announcement cite protocol items by ordinal in contract_ref.

Related Issue(s)

Closes #2576

Type of Change

Select all that apply:

Code & Documentation:

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality)
• Breaking change (fix or feature causing existing functionality to change)
• Documentation update

Infrastructure & Configuration:

• GitHub Actions workflow
• Linting configuration (markdown, PowerShell, etc.)
• Security configuration
• DevContainer configuration
• Dependency update

AI Artifacts:

• Reviewed contribution with hve-builder and addressed all actionable findings
• Copilot instructions (.github/instructions/*.instructions.md)
• Copilot prompt (.github/prompts/*.prompt.md)
• Copilot agent (.github/agents/*.agent.md)
• Copilot skill (.github/skills/*/SKILL.md)
• Copilot hook (.github/hooks/*/*.json)
• Eval spec added/updated for changed AI artifacts (evals/)

Note for AI Artifact Contributors:

• Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review .github/agents/ before creating new ones.
• Skills: Must include both bash and PowerShell scripts. See Skills.
• Model Versions: Contributions MUST target models listed in the model catalog (scripts/linting/model-catalog.json) whose provider appears in providerAllowlist and whose status is ga or preview. Run npm run lint:models to validate references.
• See Agents Not Accepted and Model Version Requirements.

Other:

• Script/automation (.ps1, .sh, .py)
• Other (please describe):

Sample Prompts (for AI Artifact Contributions)

User Request:

"Create an evaluation dataset for my IT HelpDesk Bot."

Execution Flow:

The agent runs a four-phase structured interview one question at a time, gated on user confirmation of the summary before any artifact is generated. Phase 1 now captures Responsible AI risks with a stated downstream consumer and asks for distinct user populations rather than job titles alone. Phase 5 generates the Q&A pairs against the balanced distribution plus the new coverage, provenance, and synthesis rules. Phase 6 presents 5-8 representative pairs for review, returning to Phase 5 on a regeneration request. Phase 7 writes the consolidated guide and summarizes every artifact with its path.

Output Artifacts:

Three files per run under data/evaluation/:

data/evaluation/
├── datasets/
│   ├── it-helpdesk-bot-eval-dataset.json
│   └── it-helpdesk-bot-eval-dataset.csv
└── docs/
    └── it-helpdesk-bot-eval-guide.md

The guide opens as follows:

# Evaluation Guide: IT HelpDesk Bot

## Curation Notes

### Business Context

{Business problem and scenario description from interview}

### Curation Process

#### Domain Expert Review

- [ ] Q&A pairs reviewed for accuracy

## Metric Selection

### Responsible AI Risks

| Risk   | Source               | Detecting Metric |
|--------|----------------------|------------------|
| {risk} | Interview Question 5 | {metric}         |

Success Indicators:

The run reports exactly three artifact paths, all workspace-relative under data/evaluation/. The guide carries all three H2 sections. The dataset metadata block carries population_coverage, validation_status, and generation_method, while distribution still holds exactly its five difficulty keys. Every risk named in the interview appears as a row in the Responsible AI Risks table.

For detailed contribution requirements, see:

• Common Standards: docs/contributing/ai-artifacts-common.md - Shared standards for XML blocks, markdown quality, RFC 2119, validation, and testing
• Agents: docs/contributing/custom-agents.md - Agent configurations with tools and behavior patterns
• Prompts: docs/contributing/prompts.md - Workflow-specific guidance with template variables
• Instructions: docs/contributing/instructions.md - Technology-specific standards with glob patterns
• Skills: docs/contributing/skills.md - Task execution utilities with cross-platform scripts

Testing

Automated validation:

Command	Result
npm run validate:local	Passed (26-lane aggregate, exit 0)
npm run spell-check	Passed (901 files, 0 issues)
npm run lint:md-links	Passed
npm run lint:tables	Passed
npm run lint:md	Passed (531 files, 0 issues)
npm run lint:frontmatter	Passed (840 files, 0 errors, 0 warnings)
npm run lint:yaml	Passed (65 workflow files)
npm run lint:asset-docs	Passed (257 assets, 0 errors; the 256 warnings are pre-existing repository-wide stub-placeholder notices)
npm run lint:models	Passed (27 model references, all valid)
npm run docs:generate:check	Ran; the single reported drift reproduced on a clean tree with both changed files stashed
npm run validate:docs	Not run (N/A - no docs/ changes)
npm run ci:eval:lint:vally	Pending CI (CI-owned lane)

lint:tables is the lane that reaches the changed agent file, because it enumerates with git ls-files -- '*.md'; the lint:md glob excludes .github/agents/**.

Diff-based assessment:

• Verified against the diff that distribution retains exactly easy, grounding_source_checks, hard, negative, and safety, and that the difficulty field retains exactly those five permitted values.
• Verified that exactly one expectation was added and that dataset-schema-conformance, distribution-floor, dual-format-dataset-output, interview-driven-flow, phase-transition-announcement, and no-source-modifications are unchanged.
• Swept the repository for residual references to the three retired document filenames. The only production matches are a "notes": "optional-curation-notes" JSON schema example value and unrelated prose in a different skill.
• Confirmed that no script, downstream agent, or CI schema reads the generated documents, and that the graded stimulus at evals/agent-behavior/stimuli/eval-dataset-creator.yml carries no documentation assertion and therefore needed no update.

Security analysis: no customer data, secrets, or credentials appear in the diff; no non-compliant language; no unintended files; all referenced paths exist.

Manual testing: not performed. The agent was not run end to end against a live interview.

Checklist

Required Checks

• Documentation is updated (if applicable)
• Files follow existing naming conventions
• Changes are backwards compatible (if applicable)
• Tests added for new functionality (if applicable)

AI Artifact Contributions

• Used hve-builder review mode to review contribution
• Addressed all actionable findings from the hve-builder review
• Verified contribution follows common standards and type-specific requirements

Required Local Checks

The following local-safe validation commands must pass before merging:

• Local validation aggregate: npm run validate:local
• Documentation validation (if docs changed): npm run validate:docs (N/A - no docs/ changes)
• Spell checking: npm run spell-check
• Link validation: npm run lint:md-links

Security Considerations

• This PR does not contain any sensitive or NDA information
• Any new dependencies have been reviewed for security issues (N/A - no dependency changes)
• Security-related scripts follow the principle of least privilege (N/A - no security scripts modified)

Additional Notes

The eval expectations file does not parse as strict YAML under either PyYAML or the repository's powershell-yaml module. That is not a regression: the committed baseline on main fails identically, and the backtick-leading sequence items introduced here match a convention already committed in four sibling expectation files. Separately, scripts/evals/Test-EvalSpec.ps1 explicitly excludes /agent-behavior/expectations/ from spec discovery and scripts/evals/Build-AgentBehaviorSpec.ps1 reads only stimuli/*.yml, so it is worth confirming which lane, if any, grades this file class.

Follow-up Tasks

• Establish what consumes evals/agent-behavior/expectations/*.yml and whether the agent-to-expectations contract is graded anywhere, so future work states the correct validation ownership for this file class.
• Behavior-test the agent once these edits land, confirming the new metadata fields are populated rather than echoed as placeholders.
• Address the pre-existing generated-docs drift in docs/reference/README.md, which reproduces on a clean tree and is unrelated to this change.

♻️ - Generated by Copilot

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.91%. Comparing base (d6b82a0) to head (798b2bd).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2578      +/-   ##
==========================================
+ Coverage   82.85%   82.91%   +0.06%     
==========================================
  Files         166      152      -14     
  Lines       22508    22142     -366     
  Branches       29        0      -29     
==========================================
- Hits        18648    18359     -289     
+ Misses       3857     3783      -74     
+ Partials        3        0       -3     

Flag	Coverage Δ
docusaurus	?
pester	86.22% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 15 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Eval Execution

✅ Status: Passed — no merge-blocking failures (1 advisory assertion failure(s) present)

• Artifacts evaluated: 1
• Specs run: 1
• Assertions passed: 4
• Assertions failed (blocking): 0
• Assertions failed (advisory): 1
• Failed specs (merge-blocking): 0

Artifact	Kind	Status	Specs	Passed	Failed (blocking)	Failed (advisory)
eval-dataset-creator	agent	⚠️ advisory-fail	1	4	0	1

Legend — ✅ clean · ⚠️ advisory failures only (non-blocking) · ⏭️ skipped · ❌ merge-blocking failure

Only Failed specs (merge-blocking) gates this PR. Advisory assertion failures are signal-quality checks captured during iteration; review them, but they do not block merge and may be acceptable.

[Katrien De Graeve (katriendg)]

Thanks for consolidating the eval-dataset-creator docs and closing out the RAI gaps in one pass — the heading demotion, filename repointing, and RAI additions all check out cleanly against the diff.

One thing to fix before merge: the three new failure_modes entries in the rai-metadata-completeness expectation (evals/agent-behavior/expectations/eval-dataset-creator.expectations.yml, ~lines 102–106) start with an unquoted backtick, which is invalid YAML (backtick is a reserved indicator character). Confirmed with both PyYAML and js-yaml — the file fails to parse at those lines. This directory is currently excluded from Test-EvalSpec.ps1's validation glob so CI is green, but the file's own header says it feeds a future automated rewrite pass, so it's worth fixing now. Quoting or (better) rewording those three bullets to match the file's own verb-led convention resolves both this and a related Medium-severity style note in the review.

Approving so you can get some of these closed in the conversation, and merge.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details
npm/js-yaml	4.3.1	🟢 6.1	Details Check Score Reason Packaging ⚠️ -1 packaging workflow not detected Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Maintained 🟢 10 30 commit(s) and 28 issue activity found in the last 90 days -- score normalized to 10 Code-Review ⚠️ 1 Found 3/30 approved changesets -- score normalized to 1 Token-Permissions 🟢 10 GitHub workflow tokens follow principle of least privilege Binary-Artifacts 🟢 10 no binaries found in the repo CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Pinned-Dependencies 🟢 3 dependency not pinned by hash detected -- score normalized to 3 Fuzzing 🟢 10 project is fuzzed License 🟢 10 license file detected Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches Security-Policy 🟢 4 security policy file detected SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
npm/js-yaml	4.3.1	🟢 6.1	Details Check Score Reason Packaging ⚠️ -1 packaging workflow not detected Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Maintained 🟢 10 30 commit(s) and 28 issue activity found in the last 90 days -- score normalized to 10 Code-Review ⚠️ 1 Found 3/30 approved changesets -- score normalized to 1 Token-Permissions 🟢 10 GitHub workflow tokens follow principle of least privilege Binary-Artifacts 🟢 10 no binaries found in the repo CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Pinned-Dependencies 🟢 3 dependency not pinned by hash detected -- score normalized to 3 Fuzzing 🟢 10 project is fuzzed License 🟢 10 license file detected Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches Security-Policy 🟢 4 security policy file detected SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
npm/js-yaml	3.15.1	🟢 6.1	Details Check Score Reason Packaging ⚠️ -1 packaging workflow not detected Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Maintained 🟢 10 30 commit(s) and 28 issue activity found in the last 90 days -- score normalized to 10 Code-Review ⚠️ 1 Found 3/30 approved changesets -- score normalized to 1 Token-Permissions 🟢 10 GitHub workflow tokens follow principle of least privilege Binary-Artifacts 🟢 10 no binaries found in the repo CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Pinned-Dependencies 🟢 3 dependency not pinned by hash detected -- score normalized to 3 Fuzzing 🟢 10 project is fuzzed License 🟢 10 license file detected Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches Security-Policy 🟢 4 security policy file detected SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0

Scanned Files

• docs/docusaurus/package-lock.json
• package-lock.json