chore: trigger NVSkills CI for cuopt-install

skills/cuopt-install/BENCHMARK.md

@@ -1,82 +1,102 @@
-# Evaluation Report
+# Skill Benchmark: cuopt-install
 
-Evaluation of the `cuopt-install` skill before publication through NVSkills-Eval.
+> ✅ **Overall verdict: PASS — Recommended for publication**
 
-This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.
+## Publication Recommendation
+
+Recommended for publication based on the completed evaluation evidence in this report.
 
-## Evaluation Summary
+## Evaluation Metadata
 
 - Skill: `cuopt-install`
-- Evaluation date: 2026-06-26
-- NVSkills-Eval profile: `external`
-- Environment: `astra-sandbox`
-- Dataset: 1 evaluation tasks
+- Evaluation date: 2026-08-05
+- Evaluator version: `1.0.0`
+- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)
+- Tasks: 1 evaluation tasks (1 positive)
+- Dataset digest: `sha256:75bb16e95e852c956da6f5af155ac6936922740ae833f71436ddcb715da4b6df` (skill-evaluator-dataset-snapshot/1)
 - Attempts per task: 1
-- Pass threshold: 50%
-- Overall verdict: PASS
+- Environment: `k8s-sandbox`
+- Tier 3 evidence: required for publication
 
-## Agents Used
+Each task attempt ran in its own isolated sandbox pod.
 
-- `claude-code`
-- `codex`
+## What This Report Answers
 
-## Metrics Used
+The three-tier evaluation checks whether the skill:
 
-Reported benchmark dimensions:
+- is safe to use;
+- produces correct answers;
+- is discovered and activated when needed;
+- helps the agent complete the user's goal and expected workflow; and
+- avoids wasted skill and tool usage.
 
-- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
-- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
-- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
-- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
-- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.
+## Results at a Glance
 
-Underlying evaluation signals used in this run:
+| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
+|---|---:|---:|
+| Overall | 64% → 100% (+36 points) | 69% → 91% (+22 points) |
+| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) |
+| Correctness | 100% → 100% (±0 points) | 100% → 100% (±0 points) |
+| Discoverability | 50% → 100% (+50 points) | 50% → 94% (+44 points) |
+| Effectiveness | 70% → 100% (+30 points) | 95% → 100% (+5 points) |
+| Efficiency | 0% → 100% (+100 points) | 0% → 62% (+62 points) |
 
-- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
-- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
-- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
-- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
-- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
-- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.
-- `token_efficiency` (Token Efficiency): compares token usage with and without the skill.
+**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points.
 
-## Test Tasks
+Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline.
 
-The benchmark dataset contained 1 evaluation tasks:
+## Tier Status
 
-- Positive tasks: 1 tasks where the skill was expected to activate.
-- Negative tasks: 0 tasks where no skill was expected.
-- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.
+| Tier | Purpose | Status | Evidence |
+|---|---|---|---|
+| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 5 finding(s) |
+| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded |
+| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 1 task(s) |
 
-Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.
+## Findings and Observations
 
-## Results
+<details>
+<summary>Show detailed findings and successful checks</summary>
 
-| Dimension | Num | `claude-code` | `codex` |
-|---|---:|---:|---:|
-| Security | 1 | 100% (+0%) | 100% (+0%) |
-| Correctness | 1 | 100% (+70%) | 97% (+22%) |
-| Discoverability | 1 | 100% (+100%) | 97% (+72%) |
-| Effectiveness | 1 | 100% (+74%) | 100% (+6%) |
-| Efficiency | 1 | 95% (+67%) | 96% (+69%) |
+- **MEDIUM** SCHEMA/frontmatter_field_placement: Root field 'version' is ignored; use 'metadata.version' (`skills/cuopt-install/SKILL.md`)
+- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-install/SKILL.md`)
+- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-install/SKILL.md`)
+- **LOW** SCHEMA/unexpected_file: Unexpected 'benchmark' in skill root (`skills/cuopt-install/benchmark`)
+- **LOW** SCHEMA/author_format: Author must be of the form 'Name <email@host>' (`skills/cuopt-install/SKILL.md`)
 
-Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.
+</details>
 
-## Tier 1: Static Validation Summary
+## Scoring Methodology
 
-Tier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 4 total findings.
+<details>
+<summary>Show dimension definitions, source signals, and thresholds</summary>
 
-Top findings:
+| Dimension | Question | Scored signals |
+|---|---|---|
+| Security | Is it safe to use? | `security` (100%) |
+| Correctness | Is the answer correct? | `accuracy` (100%) |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
+| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) |
 
-- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-install/SKILL.md`)
-- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-install/SKILL.md`)
-- LOW SCHEMA/unexpected_file: Unexpected 'benchmark' in skill root (`skills/cuopt-install/benchmark`)
-- LOW SCHEMA/author_format: Author must be of the form 'Name <email@host>' (`skills/cuopt-install/SKILL.md`)
+- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
+- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
+- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
+- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
+- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
+- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.
 
-## Tier 2: Deduplication Summary
+Signals present in this run:
 
-This tier was not run or did not produce findings in this report.
+- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution` (Skill Execution): whether the expected skill was found and executed.
+- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use.
+- `accuracy` (Accuracy): final-answer correctness against the reference answer.
+- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
+- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
 
-## Publication Recommendation
+</details>
+
+## Freshness
 
-The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.
+Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.

skills/cuopt-install/SKILL.md

@@ -14,6 +14,7 @@ metadata:
 ---
 
 
+
 # cuOpt Install (user)
 
 Install cuOpt to *use* it from Python, C, or as a REST server. For building cuOpt from source to contribute or modify it, see `cuopt-developer`.

skills/cuopt-install/evals/evals.json

@@ -9,5 +9,79 @@
       "Uses the nvidia/cuopt Docker image tagged for CUDA 12 (e.g. nvidia/cuopt:latest-cuda12.9-py3.13), not a fabricated nvcr.io/* path",
       "docker run command includes --gpus all and -p 8000:8000"
     ]
+  },
+  {
+    "id": "inst-eval-002-cuda-suffix-must-match",
+    "question": "nvidia-smi on my box reports CUDA 13. Which cuOpt Python package do I pip install, and does it matter which one I pick?",
+    "expected_skill": "cuopt-install",
+    "expected_script": null,
+    "ground_truth": "The agent installs the cu13-suffixed package (cuopt-cu13) and explains that the package CUDA suffix must match the runtime CUDA — installing the cu12 package on a CUDA 13 runtime is a mismatch. It includes the NVIDIA package index in the command (pip install --extra-index-url=https://pypi.nvidia.com cuopt-cu13), since cuOpt wheels are not on public PyPI without it.",
+    "expected_behavior": [
+      "Selects the cu13-suffixed package to match the reported CUDA major version",
+      "Includes --extra-index-url=https://pypi.nvidia.com in the pip command",
+      "Explains that the package CUDA suffix must match the installed CUDA runtime"
+    ]
+  },
+  {
+    "id": "inst-eval-003-python-implies-c-not-reverse",
+    "question": "I already pip installed the cuOpt Python package. Do I need to install anything else to get the C headers and shared library? And if I later only want the C API without Python, what changes?",
+    "expected_skill": "cuopt-install",
+    "expected_script": null,
+    "ground_truth": "The agent explains the one-way dependency: cuopt-cuXX depends on libcuopt-cuXX, so installing the Python package already brings in the C library and headers — nothing further is needed. The reverse does not hold: installing libcuopt-cuXX on its own provides the C API but does not install the Python bindings. For a C-only setup, install libcuopt-cuXX standalone.",
+    "expected_behavior": [
+      "States that the Python package already pulls in the C library and headers as a dependency, so no extra install is needed",
+      "States that the reverse is not true — libcuopt-cuXX alone does not provide the Python API",
+      "Names libcuopt-cuXX as the standalone C-API package"
+    ]
+  },
+  {
+    "id": "inst-eval-004-unsupported-gpu",
+    "question": "I have a GTX 1080 in this machine. Can I run cuOpt on it?",
+    "expected_skill": "cuopt-install",
+    "expected_script": null,
+    "ground_truth": "The agent answers no. cuOpt requires an NVIDIA GPU with Compute Capability 7.0 or higher (Volta or newer); the GTX 10xx series is Pascal and falls below that threshold. It names examples of supported hardware (V100, A100, H100, RTX 20xx/30xx/40xx) and does not offer a workaround or installation command that would appear to make the unsupported GPU work.",
+    "expected_behavior": [
+      "States that the GPU is not supported because cuOpt requires Compute Capability >= 7.0 (Volta or newer)",
+      "Identifies the GTX 10xx series as Pascal and therefore below the requirement",
+      "Names supported alternatives rather than suggesting a workaround for the unsupported GPU"
+    ]
+  },
+  {
+    "id": "inst-eval-005-underspecified-install-request",
+    "question": "Install cuOpt for me.",
+    "expected_skill": "cuopt-install",
+    "expected_script": null,
+    "ground_truth": "The request is underspecified, so the agent asks the required questions before giving a command: which interface (Python, C, or REST server), which CUDA version is installed (checked with nvcc --version or nvidia-smi), which package manager is preferred (pip, conda, or Docker), and what the target environment is (local GPU machine, cloud instance, container/Kubernetes, or a remote host without a local GPU). It does not guess a default and emit an install command, and it does not run the install itself.",
+    "expected_behavior": [
+      "Asks which interface is wanted — Python, C, or REST server",
+      "Asks which CUDA version is installed, referencing nvcc --version or nvidia-smi",
+      "Asks whether pip, conda, or Docker is preferred",
+      "Does not silently assume defaults and emit a single install command"
+    ]
+  },
+  {
+    "id": "inst-eval-006-no-pip-and-conda-together",
+    "question": "I installed cuOpt with conda a while back. Should I also pip install it so I definitely have the latest?",
+    "expected_skill": "cuopt-install",
+    "expected_script": null,
+    "ground_truth": "The agent advises against it: pip and conda installs of cuOpt are alternatives, not complements. Running the second one overrides the first and can leave a mismatched CUDA or package state. The agent tells the user to pick a single package manager and, if they want a newer build, to upgrade within the manager they already use rather than layering pip on top of conda.",
+    "expected_behavior": [
+      "Says not to install via both pip and conda",
+      "Explains that the second install overrides the first and risks a CUDA/package mismatch",
+      "Recommends upgrading within the already-chosen package manager instead"
+    ]
+  },
+  {
+    "id": "inst-eval-007-module-not-found-triage",
+    "question": "I installed cuOpt but `import cuopt` fails with ModuleNotFoundError: No module named 'cuopt'. How do I work out what went wrong?",
+    "expected_skill": "cuopt-install",
+    "expected_script": null,
+    "ground_truth": "The agent gives a triage sequence rather than a single guess: check that the package is actually present with pip list | grep cuopt, confirm which interpreter is being used with which python (a common cause is installing into a different environment than the one running the import), and reinstall with the correct --extra-index-url=https://pypi.nvidia.com if the package is missing. It also notes that installing only libcuopt-cuXX gives the C library without the Python bindings, which produces this same error.",
+    "expected_behavior": [
+      "Suggests checking pip list | grep cuopt to confirm the package is installed",
+      "Suggests checking which python to catch a wrong-environment mismatch",
+      "Mentions reinstalling with the NVIDIA extra index URL",
+      "Notes that a libcuopt-only install does not provide the Python module"
+    ]
   }
 ]