chore: trigger NVSkills CI for cuopt-install - #1672
Open
ramakrishnap-nv wants to merge 6 commits into
Open
Conversation
Collaborator
Author
|
/nvskills-ci |
📝 WalkthroughWalkthroughThe cuOpt install skill adds installation evaluation cases and refreshes its benchmark report, skill-card metadata, documentation spacing, and Sigstore signature bundle. ChangescuOpt install skill refresh
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
ramakrishnap-nv
added
improvement
CI Test Summary⏭️ All 5 test job(s) skipped. |
Collaborator
Author
|
/nvskills-ci |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/cuopt-install/BENCHMARK.md`:
- Around line 74-87: Update the benchmark scoring documentation near the
dimension bands and overall verdict to define Overall as the arithmetic mean of
the five skill-assisted dimension scores, including the rounding rule and score
basis. State that the verdict gate uses these skill-assisted scores rather than
baseline values, while preserving the existing threshold and verdict rules.
- Around line 15-18: Update the benchmark report metadata in BENCHMARK.md to
identify the exact evaluated input file and task ID, specifically distinguishing
the Tier 3 dataset from the inst-eval-001-docker-server task. Reconcile the
Dataset digest and task count with the actual inputs evaluated, recording both
dataset details if both sources were used.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 71a63a7f-8e5b-40c8-83f9-3f5aaab8ee84
📒 Files selected for processing (3)
skills/cuopt-install/BENCHMARK.mdskills/cuopt-install/skill-card.mdskills/cuopt-install/skill.oms.sig
Comment on lines
+15
to
+18
| - Tasks: 1 evaluation tasks (1 positive) | ||
| - Dataset digest: `sha256:75bb16e95e852c956da6f5af155ac6936922740ae833f71436ddcb715da4b6df` (skill-evaluator-dataset-snapshot/1) | ||
| - Attempts per task: 1 | ||
| - Pass threshold: 50% | ||
| - Overall verdict: PASS | ||
| - Environment: `k8s-sandbox` |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
for name in (
"skills/cuopt-install/benchmark/evals.json",
"skills/cuopt-install/evals/evals.json",
):
value = json.loads(Path(name).read_text())
records = value if isinstance(value, list) else [value]
print(name)
for record in records:
if isinstance(record, dict):
print(" id:", record.get("id"))
PY
rg -n -C 3 \
-e 'install-001-required-questions|inst-eval-001-docker-server|benchmark/evals\.json|evals/evals\.json|Tasks:|evaluation task' \
skills/cuopt-install/BENCHMARK.md \
skills/cuopt-install/skill-card.mdRepository: NVIDIA/cuopt
Length of output: 1958
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- BENCHMARK.md ---'
cat -n skills/cuopt-install/BENCHMARK.md
printf '%s\n' '--- benchmark/evals.json records ---'
python3 - <<'PY'
import json
from pathlib import Path
for name in (
"skills/cuopt-install/benchmark/evals.json",
"skills/cuopt-install/evals/evals.json",
):
print(f"\n{name}")
value = json.loads(Path(name).read_text())
records = value if isinstance(value, list) else [value]
for record in records:
print(json.dumps(record, sort_keys=True, indent=2))
PY
printf '%s\n' '--- related benchmark metadata and references ---'
rg -n -C 4 \
-e '75bb16e|skill-evaluator-dataset-snapshot|install-001-required-questions|inst-eval-001-docker-server|benchmark/evals\.json|evals/evals\.json|dataset|digest|Tier 3|Overall|Freshness' \
skills/cuopt-install .github 2>/dev/null || trueRepository: NVIDIA/cuopt
Length of output: 29978
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- skill-card evaluation section ---'
cat -n skills/cuopt-install/skill-card.md | sed -n '40,90p'
printf '%s\n' '--- repository references to the two evaluation inputs ---'
rg -n -C 5 \
-e 'skills/cuopt-install/(benchmark/evals\.json|evals/evals\.json)' \
-e 'benchmark/evals\.json|evals/evals\.json' \
-e 'inst-eval-001-docker-server|install-001-required-questions' \
--glob '!skills/cuopt-install/benchmark/evals.json' \
--glob '!skills/cuopt-install/evals/evals.json' \
.
printf '%s\n' '--- candidate dataset digests ---'
python3 - <<'PY'
import hashlib
from pathlib import Path
for name in (
"skills/cuopt-install/benchmark/evals.json",
"skills/cuopt-install/evals/evals.json",
):
data = Path(name).read_bytes()
print(name)
print(" raw sha256:", hashlib.sha256(data).hexdigest())
print(" normalized-json sha256:",
hashlib.sha256(
__import__("json").dumps(
__import__("json").loads(data),
sort_keys=True,
separators=(",", ":"),
).encode()
).hexdigest())
PYRepository: NVIDIA/cuopt
Length of output: 3080
Record the Tier 3 dataset and task ID.
skills/cuopt-install/benchmark/evals.json contains 15 tasks, while skills/cuopt-install/evals/evals.json contains inst-eval-001-docker-server. The report lists one task but identifies neither input nor task ID. Add the exact input file and task ID, and reconcile the dataset digest if both inputs were evaluated.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/cuopt-install/BENCHMARK.md` around lines 15 - 18, Update the benchmark
report metadata in BENCHMARK.md to identify the exact evaluated input file and
task ID, specifically distinguishing the Tier 3 dataset from the
inst-eval-001-docker-server task. Reconcile the Dataset digest and task count
with the actual inputs evaluated, recording both dataset details if both sources
were used.
Comment on lines
+74
to
+87
| | Dimension | Question | Scored signals | | ||
| |---|---|---| | ||
| | Security | Is it safe to use? | `security` (100%) | | ||
| | Correctness | Is the answer correct? | `accuracy` (100%) | | ||
| | Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | | ||
| | Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | | ||
| | Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | | ||
|
|
||
| - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-install/SKILL.md`) | ||
| - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-install/SKILL.md`) | ||
| - LOW SCHEMA/unexpected_file: Unexpected 'benchmark' in skill root (`skills/cuopt-install/benchmark`) | ||
| - LOW SCHEMA/author_format: Author must be of the form 'Name <email@host>' (`skills/cuopt-install/SKILL.md`) | ||
| - Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. | ||
| - Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. | ||
| - Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. | ||
| - The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. | ||
| - Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). | ||
| - Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Document the overall-score formula and gate input.
The report defines dimension thresholds and the verdict gate, but it does not define how Overall is derived from the five dimensions or whether the thresholds use baseline or skill-assisted scores. The displayed values imply an arithmetic mean, and both baseline efficiency values are 0%; without this rule, readers cannot reproduce the Overall values or the PASS decision. Add the aggregation formula, rounding rule, and explicit skill-assisted-score basis.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/cuopt-install/BENCHMARK.md` around lines 74 - 87, Update the benchmark
scoring documentation near the dimension bands and overall verdict to define
Overall as the arithmetic mean of the five skill-assisted dimension scores,
including the rounding rule and score basis. State that the verdict gate uses
these skill-assisted scores rather than baseline values, while preserving the
existing threshold and verdict rules.
Collaborator
Author
|
/ok to test 751a554 |
Collaborator
Author
|
/ok to test 1495865 |
Collaborator
Author
|
/ok to test d22f2fd |
A single eval gives the NVSkills Tier 3 live-agent evaluation almost no signal about the skill. Add six cases covering the areas the skill actually documents: CUDA suffix matching and the NVIDIA package index, the one-way cuopt-cuXX -> libcuopt-cuXX dependency, the Compute Capability 7.0 hardware floor, asking the required questions on an underspecified install request, the pip-or-conda (not both) rule, and ModuleNotFoundError triage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/cuopt-install/evals/evals.json`:
- Around line 50-60: Update the expected_behavior array for
inst-eval-005-underspecified-install-request to explicitly require asking about
the target environment, covering local GPU, cloud instance,
container/Kubernetes, or remote host without a local GPU, alongside the existing
interface, CUDA version, and package-manager requirements.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 30d036f5-46bd-4605-b6b3-96f986a03354
📒 Files selected for processing (1)
skills/cuopt-install/evals/evals.json
Comment on lines
+50
to
+60
| "id": "inst-eval-005-underspecified-install-request", | ||
| "question": "Install cuOpt for me.", | ||
| "expected_skill": "cuopt-install", | ||
| "expected_script": null, | ||
| "ground_truth": "The request is underspecified, so the agent asks the required questions before giving a command: which interface (Python, C, or REST server), which CUDA version is installed (checked with nvcc --version or nvidia-smi), which package manager is preferred (pip, conda, or Docker), and what the target environment is (local GPU machine, cloud instance, container/Kubernetes, or a remote host without a local GPU). It does not guess a default and emit an install command, and it does not run the install itself.", | ||
| "expected_behavior": [ | ||
| "Asks which interface is wanted — Python, C, or REST server", | ||
| "Asks which CUDA version is installed, referencing nvcc --version or nvidia-smi", | ||
| "Asks whether pip, conda, or Docker is preferred", | ||
| "Does not silently assume defaults and emit a single install command" | ||
| ] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Require the environment clarification in expected_behavior.
Line 54 requires the agent to ask about the target environment. Lines 55-59 do not assert that requirement. The evaluation can pass a response that omits a required question.
Proposed fix
"expected_behavior": [
"Asks which interface is wanted — Python, C, or REST server",
"Asks which CUDA version is installed, referencing nvcc --version or nvidia-smi",
"Asks whether pip, conda, or Docker is preferred",
+ "Asks about the target environment, such as a local GPU machine, cloud instance, container, or remote host",
"Does not silently assume defaults and emit a single install command"
]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| "id": "inst-eval-005-underspecified-install-request", | |
| "question": "Install cuOpt for me.", | |
| "expected_skill": "cuopt-install", | |
| "expected_script": null, | |
| "ground_truth": "The request is underspecified, so the agent asks the required questions before giving a command: which interface (Python, C, or REST server), which CUDA version is installed (checked with nvcc --version or nvidia-smi), which package manager is preferred (pip, conda, or Docker), and what the target environment is (local GPU machine, cloud instance, container/Kubernetes, or a remote host without a local GPU). It does not guess a default and emit an install command, and it does not run the install itself.", | |
| "expected_behavior": [ | |
| "Asks which interface is wanted — Python, C, or REST server", | |
| "Asks which CUDA version is installed, referencing nvcc --version or nvidia-smi", | |
| "Asks whether pip, conda, or Docker is preferred", | |
| "Does not silently assume defaults and emit a single install command" | |
| ] | |
| "id": "inst-eval-005-underspecified-install-request", | |
| "question": "Install cuOpt for me.", | |
| "expected_skill": "cuopt-install", | |
| "expected_script": null, | |
| "ground_truth": "The request is underspecified, so the agent asks the required questions before giving a command: which interface (Python, C, or REST server), which CUDA version is installed (checked with nvcc --version or nvidia-smi), which package manager is preferred (pip, conda, or Docker), and what the target environment is (local GPU machine, cloud instance, container/Kubernetes, or a remote host without a local GPU). It does not guess a default and emit an install command, and it does not run the install itself.", | |
| "expected_behavior": [ | |
| "Asks which interface is wanted — Python, C, or REST server", | |
| "Asks which CUDA version is installed, referencing nvcc --version or nvidia-smi", | |
| "Asks whether pip, conda, or Docker is preferred", | |
| "Asks about the target environment, such as a local GPU machine, cloud instance, container, or remote host", | |
| "Does not silently assume defaults and emit a single install command" | |
| ] |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/cuopt-install/evals/evals.json` around lines 50 - 60, Update the
expected_behavior array for inst-eval-005-underspecified-install-request to
explicitly require asking about the target environment, covering local GPU,
cloud instance, container/Kubernetes, or remote host without a local GPU,
alongside the existing interface, CUDA version, and package-manager
requirements.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trivial blank-line addition after SKILL.md frontmatter to trigger NVSkills CI — fetch updated skill card and benchmark for
cuopt-install.