From fd2951f64c58ed042235cb762cb2a25e2b4ce3f6 Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Wed, 5 Aug 2026 15:32:38 -0500 Subject: [PATCH 1/4] chore: trigger NVSkills CI for cuopt-routing-api-python --- skills/cuopt-routing-api-python/SKILL.md | 1 + 1 file changed, 1 insertion(+) diff --git a/skills/cuopt-routing-api-python/SKILL.md b/skills/cuopt-routing-api-python/SKILL.md index 9cf56e72a..0fbf0e7bf 100644 --- a/skills/cuopt-routing-api-python/SKILL.md +++ b/skills/cuopt-routing-api-python/SKILL.md @@ -15,6 +15,7 @@ metadata: + # cuOpt Routing — Python API This skill is **Python only**. Routing has no C API in cuOpt. From bf0fe85a860a6b2fdcb1c6935a78e2a0e16a2d75 Mon Sep 17 00:00:00 2001 From: nvskills-svc-account Date: Wed, 5 Aug 2026 22:12:19 +0000 Subject: [PATCH 2/4] Attach NVSkills validation signatures Signed-off-by: nvskills-svc-account --- skills/cuopt-routing-api-python/BENCHMARK.md | 124 ++++++++++-------- skills/cuopt-routing-api-python/skill-card.md | 58 ++++---- skills/cuopt-routing-api-python/skill.oms.sig | 2 +- 3 files changed, 102 insertions(+), 82 deletions(-) diff --git a/skills/cuopt-routing-api-python/BENCHMARK.md b/skills/cuopt-routing-api-python/BENCHMARK.md index 5384e7ef3..3674d5a2f 100644 --- a/skills/cuopt-routing-api-python/BENCHMARK.md +++ b/skills/cuopt-routing-api-python/BENCHMARK.md @@ -1,80 +1,100 @@ -# Evaluation Report +# Skill Benchmark: cuopt-routing-api-python -Evaluation of the `cuopt-routing-api-python` skill before publication through NVSkills-Eval. +> ✅ **Overall verdict: PASS — Recommended for publication** -This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use. +## Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. -## Evaluation Summary +## Evaluation Metadata - Skill: `cuopt-routing-api-python` -- Evaluation date: 2026-06-29 -- NVSkills-Eval profile: `external` -- Environment: `astra-sandbox` -- Dataset: 1 evaluation tasks +- Evaluation date: 2026-08-05 +- Evaluator version: `1.0.0` +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) +- Tasks: 1 evaluation tasks (1 positive) +- Dataset digest: `sha256:c370a78c5e91b80550c501098bba96a8fc288318b9f9830b46bca53e0d632114` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 -- Pass threshold: 50% -- Overall verdict: PASS +- Environment: `k8s-sandbox` +- Tier 3 evidence: required for publication -## Agents Used +Each task attempt ran in its own isolated sandbox pod. -- `claude-code` -- `codex` +## What This Report Answers -## Metrics Used +The three-tier evaluation checks whether the skill: -Reported benchmark dimensions: +- is safe to use; +- produces correct answers; +- is discovered and activated when needed; +- helps the agent complete the user's goal and expected workflow; and +- avoids wasted skill and tool usage. -- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. -- Correctness: checks whether the agent follows the expected workflow and produces the correct final output. -- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant. -- Effectiveness: checks whether the agent performs measurably better with the skill than without it. -- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work. +## Results at a Glance -Underlying evaluation signals used in this run: +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 52% → 95% (+43 points) | 60% → 83% (+22 points) | +| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | +| Correctness | 100% → 100% (±0 points) | 100% → 100% (±0 points) | +| Discoverability | 0% → 100% (+100 points) | 50% → 94% (+44 points) | +| Effectiveness | 59% → 75% (+16 points) | 52% → 61% (+9 points) | +| Efficiency | 0% → 100% (+100 points) | 0% → 60% (+60 points) | -- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access. -- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow. -- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage. -- `accuracy` (Accuracy): grades final-answer correctness against the reference answer. -- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully. -- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations. -- `token_efficiency` (Token Efficiency): compares token usage with and without the skill. +**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. -## Test Tasks +Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline. -The benchmark dataset contained 1 evaluation tasks: +## Tier Status -- Positive tasks: 1 tasks where the skill was expected to activate. -- Negative tasks: 0 tasks where no skill was expected. -- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred. +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 3 finding(s) | +| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 1 task(s) | -Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases. +## Findings and Observations -## Results +
+Show detailed findings and successful checks -| Dimension | Num | `claude-code` | `codex` | -|---|---:|---:|---:| -| Security | 1 | 100% (+0%) | 100% (+0%) | -| Correctness | 1 | 100% (+70%) | 97% (+42%) | -| Discoverability | 1 | 100% (+100%) | 82% (+57%) | -| Effectiveness | 1 | 82% (+45%) | 74% (+29%) | -| Efficiency | 1 | 95% (+67%) | 72% (+45%) | +- **MEDIUM** SCHEMA/frontmatter_field_placement: Root field 'version' is ignored; use 'metadata.version' (`skills/cuopt-routing-api-python/SKILL.md`) +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-routing-api-python/SKILL.md`) +- **LOW** SCHEMA/author_format: Author must be of the form 'Name ' (`skills/cuopt-routing-api-python/SKILL.md`) -Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available. +
-## Tier 1: Static Validation Summary +## Scoring Methodology -Tier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 2 total findings. +
+Show dimension definitions, source signals, and thresholds -Top findings: +| Dimension | Question | Scored signals | +|---|---|---| +| Security | Is it safe to use? | `security` (100%) | +| Correctness | Is the answer correct? | `accuracy` (100%) | +| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | +| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | +| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | -- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-routing-api-python/SKILL.md`) -- LOW SCHEMA/author_format: Author must be of the form 'Name ' (`skills/cuopt-routing-api-python/SKILL.md`) +- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. +- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. +- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. +- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. +- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). +- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. -## Tier 2: Deduplication Summary +Signals present in this run: -This tier was not run or did not produce findings in this report. +- `security` (Security): unsafe operations, secret leakage, and unauthorized access. +- `skill_execution` (Skill Execution): whether the expected skill was found and executed. +- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use. +- `accuracy` (Accuracy): final-answer correctness against the reference answer. +- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved. +- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed. -## Publication Recommendation +
+ +## Freshness -The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change. +Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes. diff --git a/skills/cuopt-routing-api-python/skill-card.md b/skills/cuopt-routing-api-python/skill-card.md index 2b0ff05fb..ff1ce50e2 100644 --- a/skills/cuopt-routing-api-python/skill-card.md +++ b/skills/cuopt-routing-api-python/skill-card.md @@ -7,16 +7,16 @@ This skill is ready for commercial/non-commercial use.
NVIDIA
### License/Terms of Use:
-Apache-2.0
+Apache 2.0
## Use Case:
-Developers and engineers building or solving vehicle routing optimization problems (VRP, TSP, PDP) using the NVIDIA cuOpt Python API.
+Developers and engineers building or solving vehicle routing problems (VRP, TSP, PDP) using the NVIDIA cuOpt Python API.
### Deployment Geography for Use:
Global
## Requirements / Dependencies:
**Requires API Key or External Credential:** [Not Specified]
-**Credential Type(s):** [None identified]
+**Credential Type(s):** [None identified]
Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate.
@@ -25,54 +25,54 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi Mitigation: Review and scan skill before deployment.
## Reference(s):
-- [examples.md](references/examples.md)
-- [server_examples.md](references/server_examples.md)
+- [Routing Python API Examples](references/examples.md)
+- [Routing REST Server Examples](references/server_examples.md)
- [cuOpt User Guide](https://docs.nvidia.com/cuopt/user-guide/latest/introduction.html)
-- [cuopt-examples](https://github.com/NVIDIA/cuopt-examples)
+- [cuOpt Examples Repository](https://github.com/NVIDIA/cuopt-examples)
## Skill Output:
**Output Type(s):** [Code, API Calls, Configuration instructions]
-**Output Format:** [Markdown with inline Python code blocks]
+**Output Format:** [Python code with inline cudf/cuOpt API calls]
**Output Parameters:** [1D]
**Other Properties Related to Output:** [None]
## Evaluation Agents Used:
-- `claude-code`
-- `codex`
+- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`)
+- Codex (`openai/openai/gpt-5.5`)
## Evaluation Tasks:
-Evaluated against 1 evaluation task in the `external` NVSkills-Eval profile (environment: astra-sandbox).
+1 evaluation task (1 positive) in isolated k8s-sandbox pods, evaluating skill-assisted routing API guidance against baseline.
## Evaluation Metrics Used:
Reported benchmark dimensions:
-- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
-- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output.
-- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
-- Effectiveness: Checks whether the agent performs measurably better with the skill than without it.
-- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work.
+- Security: Checks for unsafe operations, secret leakage, and unauthorized access.
+- Correctness: Checks final-answer correctness against the reference answer.
+- Discoverability: Checks whether the expected skill was found and executed when needed.
+- Effectiveness: Checks whether the skill helps complete the user's goal and expected workflow (goal_accuracy 50% + behavior_check 50%).
+- Efficiency: Checks routing quality, workspace-aware skill reads, and productive tool use.
Underlying evaluation signals used in this run:
-- `security`: Checks for unsafe operations, secret leakage, and unauthorized access.
-- `skill_execution`: Verifies that the agent loaded the expected skill and workflow.
-- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage.
-- `accuracy`: Grades final-answer correctness against the reference answer.
-- `goal_accuracy`: Checks whether the overall user task completed successfully.
-- `behavior_check`: Verifies expected behavior steps, including safety expectations.
-- `token_efficiency`: Compares token usage with and without the skill.
+- `security`: Verifies absence of unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution`: Verifies whether the expected skill was found and executed.
+- `skill_efficiency`: Verifies routing quality, workspace-aware skill reads, and productive tool use.
+- `accuracy`: Verifies final-answer correctness against the reference answer.
+- `goal_accuracy`: Verifies whether the user's goal was achieved.
+- `behavior_check`: Verifies whether the expected workflow behavior was followed.
## Evaluation Results:
-| Dimension | Num | `claude-code` | `codex` | -|---|---:|---:|---:| -| Security | 1 | 100% (+0%) | 100% (+0%) | -| Correctness | 1 | 100% (+70%) | 97% (+42%) | -| Discoverability | 1 | 100% (+100%) | 82% (+57%) | -| Effectiveness | 1 | 82% (+45%) | 74% (+29%) | -| Efficiency | 1 | 95% (+67%) | 72% (+45%) | +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 52% → 95% (+43 points) | 60% → 83% (+22 points) | +| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | +| Correctness | 100% → 100% (±0 points) | 100% → 100% (±0 points) | +| Discoverability | 0% → 100% (+100 points) | 50% → 94% (+44 points) | +| Effectiveness | 59% → 75% (+16 points) | 52% → 61% (+9 points) | +| Efficiency | 0% → 100% (+100 points) | 0% → 60% (+60 points) | ## Skill Version(s):
26.10.00 (source: frontmatter)
diff --git a/skills/cuopt-routing-api-python/skill.oms.sig b/skills/cuopt-routing-api-python/skill.oms.sig index 27dfd3f0e..92dedda53 100644 --- a/skills/cuopt-routing-api-python/skill.oms.sig +++ b/skills/cuopt-routing-api-python/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiY3VvcHQtcm91dGluZy1hcGktcHl0aG9uIiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjIxNTEyZWMwZjZlZDM0OWE4YjNkMGNiZmY3ZDljOGM5MzA0MThhMzAzMjdjZmM1NDk2ZTE0OTE0NGQ2ZmRiOTEiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogImE1MzkwNDM2YmNlM2ZkZjMyMTEyZjUxZmIxZjU0Mzk2MjdjNDM2NzU1Zjg3MjRiYzlhYTVhNzhkMjlhYmZmODciLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJCRU5DSE1BUksubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjNkODg1YzAzNjAwODQ2NDI2NmFiMzc3NWZkNTczZDBhYjA0ZjdkNTU3NDkwNDI5ODlhYjFkZWQ4MTMzMTU2MGMiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiYzcwMDJhMzEyMTk4M2YzMjk0ZWZiZThjOTU0MTk0M2JmMjRjODlhMDdiZWE2YTMyMDA3Yzc3NGEyYzgwODAyMSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogImFzc2V0cy9SRUFETUUubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogImUwZDFhMWZkN2QwYWQ0ZTQ1NGQwOGY1NWRlOTFiYmVkZTc4ZmYxMjIyZDZhNTQyZDU1YWFhY2Y3MWM1YTdlNjIiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJhc3NldHMvcGRwX2Jhc2ljL1JFQURNRS5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiMTQ0ZGExZGY1ZGUyOGQ3ODVhOWI0NjdiM2QxNDQxN2U3MTZmNTMyYWM5Yjk4OTA0NmVhZjdlNGY1Mjk5YTVmZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogImFzc2V0cy9wZHBfYmFzaWMvbW9kZWwucHkiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjBhODVhZWYxYzFiZTU5OTg5ZGU0MGFhNmNlOWZlNTRlNzIwZTcwOTVmMDg2YmM2ZjY4NGYyYjNjOWRhMzM4OTQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJhc3NldHMvdnJwX2Jhc2ljL1JFQURNRS5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiNTI0NWI3NzQ2NWEyNmI2OGFlZmJhYTMyNDliNTFlZjBkYTQ1MDVmNGUxOTc0Y2Y2ZDBmNDRiMWM3OGZjODA3MCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogImFzc2V0cy92cnBfYmFzaWMvbW9kZWwucHkiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogImRiYTIwMTExOWU0ZjBjNWI3ZDdiMTdkZjljNzExZDQ0ODNlMDE4OTMzNTA4ZTM2MWVlOTI5ZTZjNDRlNTZhNGIiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJldmFscy9ldmFscy5qc29uIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI1ZWIzNTczNTU1OWQ5MzVjMjFlMjBhNDk1OTFkNTRjOWI2ZmE5MjQ3MjE3YWE0Nzc2YTVmODk4M2NiZDI3ZjgxIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9leGFtcGxlcy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiNTA4YmY0YWU4Y2I1YmM3ZTI0OWIzNzcyNjBmMTQyMWI3MGQ5ZDM0NWJiNWE2ZDE2YzZmYTBiNTZlMjU2ODI1YiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvc2VydmVyX2V4YW1wbGVzLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJmYjMxYjBmZWQ4ZDNjOGQ5ZDRiMWU0OTkxMTFkZmJlNzNhMDM3ZmU1YmEzOWY5YjQ0Njc1MTI5NzBhZTZmY2JlIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIKICAgICAgfQogICAgXSwKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXRodWIiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIsCiAgICAgICAgIi5naXRpZ25vcmUiLAogICAgICAgICIuZ2l0IgogICAgICBdLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIKICAgIH0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGQCMEbltLo1vI/GV6W/VzAVAb+tJRcudQZzHMA1PfXTUd+sfc3njINmcL7FN5Z1DcFOpQIwTeEpHNv09maxv2FO4NF9Y4Yu2rCnOW15b7KPjIL3cH8FYgFeLSG2VwfZgFuKa8OE","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiY3VvcHQtcm91dGluZy1hcGktcHl0aG9uIiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjc2ZDc1ZjQ2YmM2OTA5Y2ZjM2U1MDdlYjczMWI4ZGIwNjYyZTc2MTAyZjcyZTM4ODFjZjI3NTVmOWQzNDI4ZGMiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXRodWIiCiAgICAgIF0KICAgIH0sCiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIiwKICAgICAgICAiZGlnZXN0IjogImQyMGE4YTVkMDIxNTU5NzkwMTg4ZGRhYTllOThmY2I2MDU3OTg1NTgxNjhhNjU1ZmVjNzA5YzM4Y2VkN2RjNmMiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiLAogICAgICAgICJkaWdlc3QiOiAiZGVmMTI2NmVlY2VlOGMwYWNiNTg5NWI1NzNlMzFjZDMwNWJjMDI2ZDE3YjIwMWI4ZTc5Y2I5NDllYzQxZDlhOSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJhc3NldHMvUkVBRE1FLm1kIiwKICAgICAgICAiZGlnZXN0IjogImM3MDAyYTMxMjE5ODNmMzI5NGVmYmU4Yzk1NDE5NDNiZjI0Yzg5YTA3YmVhNmEzMjAwN2M3NzRhMmM4MDgwMjEiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiYXNzZXRzL3BkcF9iYXNpYy9SRUFETUUubWQiLAogICAgICAgICJkaWdlc3QiOiAiZTBkMWExZmQ3ZDBhZDRlNDU0ZDA4ZjU1ZGU5MWJiZWRlNzhmZjEyMjJkNmE1NDJkNTVhYWFjZjcxYzVhN2U2MiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJhc3NldHMvcGRwX2Jhc2ljL21vZGVsLnB5IiwKICAgICAgICAiZGlnZXN0IjogIjE0NGRhMWRmNWRlMjhkNzg1YTliNDY3YjNkMTQ0MTdlNzE2ZjUzMmFjOWI5ODkwNDZlYWY3ZTRmNTI5OWE1ZmQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiYXNzZXRzL3ZycF9iYXNpYy9SRUFETUUubWQiLAogICAgICAgICJkaWdlc3QiOiAiMGE4NWFlZjFjMWJlNTk5ODlkZTQwYWE2Y2U5ZmU1NGU3MjBlNzA5NWYwODZiYzZmNjg0ZjJiM2M5ZGEzMzg5NCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJhc3NldHMvdnJwX2Jhc2ljL21vZGVsLnB5IiwKICAgICAgICAiZGlnZXN0IjogIjUyNDViNzc0NjVhMjZiNjhhZWZiYWEzMjQ5YjUxZWYwZGE0NTA1ZjRlMTk3NGNmNmQwZjQ0YjFjNzhmYzgwNzAiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImRpZ2VzdCI6ICJkYmEyMDExMTllNGYwYzViN2Q3YjE3ZGY5YzcxMWQ0NDgzZTAxODkzMzUwOGUzNjFlZTkyOWU2YzQ0ZTU2YTRiIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvZXhhbXBsZXMubWQiLAogICAgICAgICJkaWdlc3QiOiAiNWViMzU3MzU1NTlkOTM1YzIxZTIwYTQ5NTkxZDU0YzliNmZhOTI0NzIxN2FhNDc3NmE1Zjg5ODNjYmQyN2Y4MSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3NlcnZlcl9leGFtcGxlcy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICI1MDhiZjRhZThjYjViYzdlMjQ5YjM3NzI2MGYxNDIxYjcwZDlkMzQ1YmI1YTZkMTZjNmZhMGI1NmUyNTY4MjViIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJkaWdlc3QiOiAiYTdhZDg3MTFmMWVjNGVkM2M1NWM5MTI2ODMwMDcwYTU2YzhhOTdjMzgxOWZmYmE2NmNjNGU5ZDMwYTE3NTY3NiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0KICAgIF0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMG0K5xCF+DxJu3YMqGv8hm/p4WixB0X95zOwA4jnra69csB5eGSylAWmKg9ikPa99gIxAKMm5098FKHiMppOLx8F6/+qGbf+MEaiWGYMpDWBNOcX7oj3OBc5n6bIpC7KbJHirg==","keyid":""}]}} \ No newline at end of file From 00a25731aa188b40cbaa6c0b27d4bc75e4285bb8 Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Tue, 11 Aug 2026 11:07:48 -0500 Subject: [PATCH 3/4] chore: expand cuopt-routing-api-python eval coverage from 1 to 7 cases A single eval gives the NVSkills Tier 3 live-agent evaluation almost no signal about the skill. Add six cases covering areas the skill documents but the existing eval does not exercise: solution status codes and the get_error_message / get_infeasible_orders triage path, add_capacity_dimension, set_pickup_delivery_pairs versus add_order_precedence, the explicit float32/int32 dtype requirement, the compute_waypoint_sequence in-place mutation gotcha, and the fact that routing has no C API. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Ramakrishna Prabhu --- .../cuopt-routing-api-python/evals/evals.json | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/skills/cuopt-routing-api-python/evals/evals.json b/skills/cuopt-routing-api-python/evals/evals.json index ee89609c8..fef47f5cb 100644 --- a/skills/cuopt-routing-api-python/evals/evals.json +++ b/skills/cuopt-routing-api-python/evals/evals.json @@ -15,5 +15,79 @@ "Mentions explicit dtypes (float32 for matrices, int32 for index series)", "Does not invent method names that are not in the skill" ] + }, + { + "id": "rt-py-eval-002-status-codes-and-triage", + "question": "routing.Solve came back and my route is empty. How do I find out what happened? What do the status values mean?", + "expected_skill": "cuopt-routing-api-python", + "expected_script": null, + "ground_truth": "The agent explains that solution.get_status() must be checked first, and gives the meaning of the values: 0 = SUCCESS, 1 = FAIL, 2 = TIMEOUT, 3 = EMPTY. For any non-zero status it directs the user to solution.get_error_message() for the reason and solution.get_infeasible_orders().to_list() to see which specific orders could not be served. It also names the usual causes from the skill's issue table — an empty solution generally means time windows are too tight or the travel times are wrong, and infeasible orders generally mean insufficient fleet or capacity. It does not invent additional status codes.", + "expected_behavior": [ + "States that solution.get_status() == 0 means success and enumerates 1 = FAIL, 2 = TIMEOUT, 3 = EMPTY", + "Directs the user to solution.get_error_message() and solution.get_infeasible_orders().to_list()", + "Links an empty solution to over-tight time windows or wrong travel times, and infeasible orders to insufficient fleet or capacity", + "Does not invent status codes beyond those documented" + ] + }, + { + "id": "rt-py-eval-003-capacity-dimension", + "question": "My vehicles have a weight limit and each order has a weight. How do I express that in the cuOpt Python routing API?", + "expected_skill": "cuopt-routing-api-python", + "expected_script": null, + "ground_truth": "The agent uses dm.add_capacity_dimension(name, demand_series, capacity_series) — a named dimension carrying the per-order demand and the per-vehicle capacity. It notes that the method is called once per capacity dimension, so multiple dimensions (e.g. weight and volume) mean multiple calls, and that the demand and capacity series should be int32. It does not invent a set_capacity or add_vehicle_capacity method.", + "expected_behavior": [ + "Names add_capacity_dimension with a dimension name, a per-order demand series, and a per-vehicle capacity series", + "States that additional capacity dimensions require additional calls", + "Mentions int32 dtype for the demand/capacity series", + "Does not invent an alternative capacity method name" + ] + }, + { + "id": "rt-py-eval-004-pickup-delivery-pairs", + "question": "I need each pickup to be served by the same vehicle that later makes the matching delivery. Which cuOpt Python routing call sets that up?", + "expected_skill": "cuopt-routing-api-python", + "expected_script": null, + "ground_truth": "The agent names dm.set_pickup_delivery_pairs(pickup_indices, delivery_indices), which takes parallel index series pairing each pickup order with its delivery order and makes the problem a PDP. It notes the indices refer to order indices, that the pairing is what enforces same-vehicle and pickup-before-delivery behaviour, and that dm.add_order_precedence(node_id=..., preceding_nodes=...) is the separate mechanism for general precedence between nodes rather than pickup/delivery pairing.", + "expected_behavior": [ + "Names set_pickup_delivery_pairs with paired pickup and delivery index series", + "Explains that the pairing is what ties a pickup and its delivery to the same vehicle in order", + "Distinguishes it from add_order_precedence, which expresses general precedence" + ] + }, + { + "id": "rt-py-eval-005-explicit-dtypes", + "question": "Do I need to care about dtypes when I build the cost matrix and order location series for cuOpt routing in Python, or will pandas/cudf defaults be fine?", + "expected_skill": "cuopt-routing-api-python", + "expected_script": null, + "ground_truth": "The agent says dtypes must be set explicitly rather than left to defaults: the cost and transit time matrices should be float32, and index-like series such as order locations and demand should be int32. It explains that relying on defaults (e.g. float64 or int64) can produce silent errors rather than a clear exception, which is why the skill calls for explicit casts such as cost_matrix.astype('float32') and cudf.Series([...], dtype='int32').", + "expected_behavior": [ + "States that float32 is expected for the cost and transit time matrices", + "States that int32 is expected for order location and demand series", + "Explains that leaving dtypes to default risks silent errors, so explicit casting is required" + ] + }, + { + "id": "rt-py-eval-006-waypoint-sequence-mutates-route-df", + "question": "I'm looping over trucks and calling compute_waypoint_sequence on the route dataframe for each one, but after the first truck my cost-matrix indices are wrong. What's going on?", + "expected_skill": "cuopt-routing-api-python", + "expected_script": null, + "ground_truth": "The agent identifies the in-place mutation: compute_waypoint_sequence replaces the location column of the route dataframe with waypoint ids in place, so the original cost-matrix indices are destroyed after the first call and every later iteration reads waypoint ids instead. The fix is to pass route_df.copy() into the call whenever the cost-matrix indices are still needed afterwards, which is exactly the per-truck loop case.", + "expected_behavior": [ + "Identifies that compute_waypoint_sequence overwrites the location column with waypoint ids in place", + "Explains that this is why later loop iterations see wrong indices", + "Gives passing route_df.copy() as the fix" + ] + }, + { + "id": "rt-py-eval-007-no-c-api-for-routing", + "question": "I'm writing a C application and want to call cuOpt's VRP solver directly from C. How do I do that?", + "expected_skill": "cuopt-routing-api-python", + "expected_script": null, + "ground_truth": "The agent states that routing has no C API in cuOpt — the routing solver is exposed through the Python API only. It redirects the user to a workable alternative: drive routing from Python, or call the cuOpt REST server over HTTP from the C application, since the server is language-agnostic. It does not fabricate C routing entry points or header names.", + "expected_behavior": [ + "States clearly that cuOpt routing has no C API and is Python-only", + "Offers the REST server as the language-agnostic route for a non-Python caller", + "Does not invent C function or header names for routing" + ] } ] From 14d70937f6d1a02003cc9336fb3c2846689c5c4b Mon Sep 17 00:00:00 2001 From: nvskills-svc-account Date: Wed, 12 Aug 2026 17:04:41 +0000 Subject: [PATCH 4/4] Attach NVSkills validation signatures Signed-off-by: nvskills-svc-account --- skills/cuopt-routing-api-python/BENCHMARK.md | 20 ++++++------- skills/cuopt-routing-api-python/skill-card.md | 28 +++++++++---------- skills/cuopt-routing-api-python/skill.oms.sig | 2 +- 3 files changed, 25 insertions(+), 25 deletions(-) diff --git a/skills/cuopt-routing-api-python/BENCHMARK.md b/skills/cuopt-routing-api-python/BENCHMARK.md index 3674d5a2f..3d3370862 100644 --- a/skills/cuopt-routing-api-python/BENCHMARK.md +++ b/skills/cuopt-routing-api-python/BENCHMARK.md @@ -9,11 +9,11 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata - Skill: `cuopt-routing-api-python` -- Evaluation date: 2026-08-05 -- Evaluator version: `1.0.0` +- Evaluation date: 2026-08-12 +- Evaluator version: `1.2.4` - Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) -- Tasks: 1 evaluation tasks (1 positive) -- Dataset digest: `sha256:c370a78c5e91b80550c501098bba96a8fc288318b9f9830b46bca53e0d632114` (skill-evaluator-dataset-snapshot/1) +- Tasks: 7 evaluation tasks (7 positive) +- Dataset digest: `sha256:e13146ac43d578682f0a3d799837f67335d277ef722feb1d28fc526709b6ba4e` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 - Environment: `k8s-sandbox` - Tier 3 evidence: required for publication @@ -34,12 +34,12 @@ The three-tier evaluation checks whether the skill: | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 52% → 95% (+43 points) | 60% → 83% (+22 points) | +| Overall | 54% → 91% (+37 points) | 60% → 86% (+26 points) | | Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | -| Correctness | 100% → 100% (±0 points) | 100% → 100% (±0 points) | -| Discoverability | 0% → 100% (+100 points) | 50% → 94% (+44 points) | -| Effectiveness | 59% → 75% (+16 points) | 52% → 61% (+9 points) | -| Efficiency | 0% → 100% (+100 points) | 0% → 60% (+60 points) | +| Correctness | 77% → 91% (+14 points) | 80% → 100% (+20 points) | +| Discoverability | 21% → 93% (+71 points) | 50% → 88% (+38 points) | +| Effectiveness | 66% → 87% (+21 points) | 65% → 86% (+21 points) | +| Efficiency | 8% → 86% (+78 points) | 7% → 56% (+49 points) | **How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. @@ -51,7 +51,7 @@ Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 |---|---|---|---| | Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 3 finding(s) | | Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded | -| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 1 task(s) | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 7 task(s) | ## Findings and Observations diff --git a/skills/cuopt-routing-api-python/skill-card.md b/skills/cuopt-routing-api-python/skill-card.md index ff1ce50e2..253d42e00 100644 --- a/skills/cuopt-routing-api-python/skill-card.md +++ b/skills/cuopt-routing-api-python/skill-card.md @@ -7,7 +7,7 @@ This skill is ready for commercial/non-commercial use.
NVIDIA
### License/Terms of Use:
-Apache 2.0
+Apache-2.0
## Use Case:
Developers and engineers building or solving vehicle routing problems (VRP, TSP, PDP) using the NVIDIA cuOpt Python API.
@@ -25,15 +25,15 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi Mitigation: Review and scan skill before deployment.
## Reference(s):
-- [Routing Python API Examples](references/examples.md)
-- [Routing REST Server Examples](references/server_examples.md)
+- [examples.md](references/examples.md)
+- [server_examples.md](references/server_examples.md)
- [cuOpt User Guide](https://docs.nvidia.com/cuopt/user-guide/latest/introduction.html)
-- [cuOpt Examples Repository](https://github.com/NVIDIA/cuopt-examples)
+- [cuOpt Examples](https://github.com/NVIDIA/cuopt-examples)
## Skill Output:
**Output Type(s):** [Code, API Calls, Configuration instructions]
-**Output Format:** [Python code with inline cudf/cuOpt API calls]
+**Output Format:** [Markdown with inline Python code blocks]
**Output Parameters:** [1D]
**Other Properties Related to Output:** [None]
@@ -44,19 +44,19 @@ Mitigation: Review and scan skill before deployment.
## Evaluation Tasks:
-1 evaluation task (1 positive) in isolated k8s-sandbox pods, evaluating skill-assisted routing API guidance against baseline.
+7 evaluation tasks (7 positive) run in isolated sandbox pods.
## Evaluation Metrics Used:
Reported benchmark dimensions:
- Security: Checks for unsafe operations, secret leakage, and unauthorized access.
- Correctness: Checks final-answer correctness against the reference answer.
- Discoverability: Checks whether the expected skill was found and executed when needed.
-- Effectiveness: Checks whether the skill helps complete the user's goal and expected workflow (goal_accuracy 50% + behavior_check 50%).
+- Effectiveness: Checks whether the skill helped complete the user's goal and expected workflow.
- Efficiency: Checks routing quality, workspace-aware skill reads, and productive tool use.
Underlying evaluation signals used in this run:
-- `security`: Verifies absence of unsafe operations, secret leakage, and unauthorized access.
-- `skill_execution`: Verifies whether the expected skill was found and executed.
+- `security`: Detects unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution`: Verifies the expected skill was found and executed.
- `skill_efficiency`: Verifies routing quality, workspace-aware skill reads, and productive tool use.
- `accuracy`: Verifies final-answer correctness against the reference answer.
- `goal_accuracy`: Verifies whether the user's goal was achieved.
@@ -67,12 +67,12 @@ Underlying evaluation signals used in this run:
## Evaluation Results:
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 52% → 95% (+43 points) | 60% → 83% (+22 points) | +| Overall | 54% → 91% (+37 points) | 60% → 86% (+26 points) | | Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | -| Correctness | 100% → 100% (±0 points) | 100% → 100% (±0 points) | -| Discoverability | 0% → 100% (+100 points) | 50% → 94% (+44 points) | -| Effectiveness | 59% → 75% (+16 points) | 52% → 61% (+9 points) | -| Efficiency | 0% → 100% (+100 points) | 0% → 60% (+60 points) | +| Correctness | 77% → 91% (+14 points) | 80% → 100% (+20 points) | +| Discoverability | 21% → 93% (+71 points) | 50% → 88% (+38 points) | +| Effectiveness | 66% → 87% (+21 points) | 65% → 86% (+21 points) | +| Efficiency | 8% → 86% (+78 points) | 7% → 56% (+49 points) | ## Skill Version(s):
26.10.00 (source: frontmatter)
diff --git a/skills/cuopt-routing-api-python/skill.oms.sig b/skills/cuopt-routing-api-python/skill.oms.sig index 92dedda53..772f6fdca 100644 --- a/skills/cuopt-routing-api-python/skill.oms.sig +++ b/skills/cuopt-routing-api-python/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiY3VvcHQtcm91dGluZy1hcGktcHl0aG9uIiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjc2ZDc1ZjQ2YmM2OTA5Y2ZjM2U1MDdlYjczMWI4ZGIwNjYyZTc2MTAyZjcyZTM4ODFjZjI3NTVmOWQzNDI4ZGMiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXRodWIiCiAgICAgIF0KICAgIH0sCiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIiwKICAgICAgICAiZGlnZXN0IjogImQyMGE4YTVkMDIxNTU5NzkwMTg4ZGRhYTllOThmY2I2MDU3OTg1NTgxNjhhNjU1ZmVjNzA5YzM4Y2VkN2RjNmMiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiLAogICAgICAgICJkaWdlc3QiOiAiZGVmMTI2NmVlY2VlOGMwYWNiNTg5NWI1NzNlMzFjZDMwNWJjMDI2ZDE3YjIwMWI4ZTc5Y2I5NDllYzQxZDlhOSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJhc3NldHMvUkVBRE1FLm1kIiwKICAgICAgICAiZGlnZXN0IjogImM3MDAyYTMxMjE5ODNmMzI5NGVmYmU4Yzk1NDE5NDNiZjI0Yzg5YTA3YmVhNmEzMjAwN2M3NzRhMmM4MDgwMjEiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiYXNzZXRzL3BkcF9iYXNpYy9SRUFETUUubWQiLAogICAgICAgICJkaWdlc3QiOiAiZTBkMWExZmQ3ZDBhZDRlNDU0ZDA4ZjU1ZGU5MWJiZWRlNzhmZjEyMjJkNmE1NDJkNTVhYWFjZjcxYzVhN2U2MiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJhc3NldHMvcGRwX2Jhc2ljL21vZGVsLnB5IiwKICAgICAgICAiZGlnZXN0IjogIjE0NGRhMWRmNWRlMjhkNzg1YTliNDY3YjNkMTQ0MTdlNzE2ZjUzMmFjOWI5ODkwNDZlYWY3ZTRmNTI5OWE1ZmQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiYXNzZXRzL3ZycF9iYXNpYy9SRUFETUUubWQiLAogICAgICAgICJkaWdlc3QiOiAiMGE4NWFlZjFjMWJlNTk5ODlkZTQwYWE2Y2U5ZmU1NGU3MjBlNzA5NWYwODZiYzZmNjg0ZjJiM2M5ZGEzMzg5NCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJhc3NldHMvdnJwX2Jhc2ljL21vZGVsLnB5IiwKICAgICAgICAiZGlnZXN0IjogIjUyNDViNzc0NjVhMjZiNjhhZWZiYWEzMjQ5YjUxZWYwZGE0NTA1ZjRlMTk3NGNmNmQwZjQ0YjFjNzhmYzgwNzAiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImRpZ2VzdCI6ICJkYmEyMDExMTllNGYwYzViN2Q3YjE3ZGY5YzcxMWQ0NDgzZTAxODkzMzUwOGUzNjFlZTkyOWU2YzQ0ZTU2YTRiIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvZXhhbXBsZXMubWQiLAogICAgICAgICJkaWdlc3QiOiAiNWViMzU3MzU1NTlkOTM1YzIxZTIwYTQ5NTkxZDU0YzliNmZhOTI0NzIxN2FhNDc3NmE1Zjg5ODNjYmQyN2Y4MSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3NlcnZlcl9leGFtcGxlcy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICI1MDhiZjRhZThjYjViYzdlMjQ5YjM3NzI2MGYxNDIxYjcwZDlkMzQ1YmI1YTZkMTZjNmZhMGI1NmUyNTY4MjViIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJkaWdlc3QiOiAiYTdhZDg3MTFmMWVjNGVkM2M1NWM5MTI2ODMwMDcwYTU2YzhhOTdjMzgxOWZmYmE2NmNjNGU5ZDMwYTE3NTY3NiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0KICAgIF0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMG0K5xCF+DxJu3YMqGv8hm/p4WixB0X95zOwA4jnra69csB5eGSylAWmKg9ikPa99gIxAKMm5098FKHiMppOLx8F6/+qGbf+MEaiWGYMpDWBNOcX7oj3OBc5n6bIpC7KbJHirg==","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiY3VvcHQtcm91dGluZy1hcGktcHl0aG9uIiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogImI4OWMwNTUzNzJmNDFjN2U5YmNmMjBkMGIzZDYwY2NhZGZmNzg1Y2VlNTM5NjU2MGM1NmU4ZTFmZjNiYThhZTAiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGlnbm9yZSIKICAgICAgXSwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJtZXRob2QiOiAiZmlsZXMiLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIKICAgIH0sCiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMjY2MWNjYzM0MjJiYWIzODlhYWQ0NmU4ZWZiOWZjY2ExN2JmOWFkZDlkYTMwYzkwNmVlMWFhNWRhNGRjOTA1NiIsCiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiZGVmMTI2NmVlY2VlOGMwYWNiNTg5NWI1NzNlMzFjZDMwNWJjMDI2ZDE3YjIwMWI4ZTc5Y2I5NDllYzQxZDlhOSIsCiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJjNzAwMmEzMTIxOTgzZjMyOTRlZmJlOGM5NTQxOTQzYmYyNGM4OWEwN2JlYTZhMzIwMDdjNzc0YTJjODA4MDIxIiwKICAgICAgICAibmFtZSI6ICJhc3NldHMvUkVBRE1FLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiZTBkMWExZmQ3ZDBhZDRlNDU0ZDA4ZjU1ZGU5MWJiZWRlNzhmZjEyMjJkNmE1NDJkNTVhYWFjZjcxYzVhN2U2MiIsCiAgICAgICAgIm5hbWUiOiAiYXNzZXRzL3BkcF9iYXNpYy9SRUFETUUubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICIxNDRkYTFkZjVkZTI4ZDc4NWE5YjQ2N2IzZDE0NDE3ZTcxNmY1MzJhYzliOTg5MDQ2ZWFmN2U0ZjUyOTlhNWZkIiwKICAgICAgICAibmFtZSI6ICJhc3NldHMvcGRwX2Jhc2ljL21vZGVsLnB5IgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMGE4NWFlZjFjMWJlNTk5ODlkZTQwYWE2Y2U5ZmU1NGU3MjBlNzA5NWYwODZiYzZmNjg0ZjJiM2M5ZGEzMzg5NCIsCiAgICAgICAgIm5hbWUiOiAiYXNzZXRzL3ZycF9iYXNpYy9SRUFETUUubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI1MjQ1Yjc3NDY1YTI2YjY4YWVmYmFhMzI0OWI1MWVmMGRhNDUwNWY0ZTE5NzRjZjZkMGY0NGIxYzc4ZmM4MDcwIiwKICAgICAgICAibmFtZSI6ICJhc3NldHMvdnJwX2Jhc2ljL21vZGVsLnB5IgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMzU2Njg0ODI2YjhlMTJkMTJlNjM1NTM4NTQ5YjRhMzQzZmUyZDMzM2NiZjZhYjdjN2RiMGI4YzZjY2FjNWYxOSIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjVlYjM1NzM1NTU5ZDkzNWMyMWUyMGE0OTU5MWQ1NGM5YjZmYTkyNDcyMTdhYTQ3NzZhNWY4OTgzY2JkMjdmODEiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvZXhhbXBsZXMubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI1MDhiZjRhZThjYjViYzdlMjQ5YjM3NzI2MGYxNDIxYjcwZDlkMzQ1YmI1YTZkMTZjNmZhMGI1NmUyNTY4MjViIiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3NlcnZlcl9leGFtcGxlcy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjcyNGIwYjdlZWFmOGUyZjExM2MxMjNhYWRhN2EwZGZhOGE4OWFiNjM4OTg2MmFjMzMyMDVlMWIxZjc0OTYzZjEiLAogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiCiAgICAgIH0KICAgIF0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQDSnNfMvdSaehE67kFgRt8yzaZByeV0HtcjceFwajtInLg2ueiArYt8kMq8AODTnoECMDnSvgMHoYmg/YhMxDVKMbWbumBXOQXBtE7TjxFCqqx+MRdtRSHylUnkndOGFaSObg==","keyid":""}]}} \ No newline at end of file