From 8c036057f3bf93a21fad96c4a5b0f2cc7a5a78f2 Mon Sep 17 00:00:00 2001 From: Anipik Date: Fri, 4 Sep 2026 10:21:57 -0700 Subject: [PATCH 1/4] docs: update eval commands from `uip or eval` to `uip eval` Eval commands moved from orchestrator-tool to a standalone eval-tool package in UiPath/cli#4025. Commands are now `uip eval ...` instead of `uip or eval ...`. Co-Authored-By: Claude Opus 4.6 (1M context) --- skills/uipath-agents/SKILL.md | 2 +- .../lowcode/evaluations/evaluate.md | 2 +- .../evaluations/orchestrator-eval-run.md | 464 ------------------ .../lowcode/eval_run/eval_run.yaml | 2 +- 4 files changed, 3 insertions(+), 467 deletions(-) diff --git a/skills/uipath-agents/SKILL.md b/skills/uipath-agents/SKILL.md index acb99bb710..da5597d557 100644 --- a/skills/uipath-agents/SKILL.md +++ b/skills/uipath-agents/SKILL.md @@ -69,7 +69,7 @@ Determine the agent mode before proceeding: | Check, validate, diagnose, or fix whether an existing coded guardrail is correctly configured (placement / scope) | Coded | [coded/capabilities/guardrails/guardrails-recommend.md](references/coded/capabilities/guardrails/guardrails-recommend.md) § Validate Mode | **fetch SDK docs first (authoritative for scope/placement)**; also fetch catalog + list for relevance/entitlement — then check correctness, actionability, and relevance — fix in place | | Embed a low-code agent inline in a flow, or wire a multi-agent solution | Low-code | [lowcode/lowcode.md](references/lowcode/lowcode.md) § Capability Registry | `lowcode/capabilities/inline-in-flow/inline-in-flow.md`, `lowcode/capabilities/process/solution-agent.md` | | Run low-code evaluations | Low-code | [lowcode/evaluations/evaluate.md](references/lowcode/evaluations/evaluate.md) | `lowcode/evaluations/evaluators.md`, `lowcode/evaluations/evaluation-sets.md`, `lowcode/evaluations/running-evaluations.md` | -| Manage runtime evals for a published Orchestrator package | Low-code | [lowcode/evaluations/orchestrator-eval-run.md](references/lowcode/evaluations/orchestrator-eval-run.md) | `uip or eval execute-and-evaluate`, evaluator/eval-set/evaluation CRUD, run results, schedules | +| Manage runtime evals for a published Orchestrator package | Low-code | [lowcode/evaluations/orchestrator-eval-run.md](references/lowcode/evaluations/orchestrator-eval-run.md) | `uip eval execute-and-evaluate`, evaluator/eval-set/evaluation CRUD, run results, schedules | | Validate, pack, publish, upload, or deploy a low-code agent | Low-code | [lowcode/lowcode.md](references/lowcode/lowcode.md) | `lowcode/project-lifecycle.md`, `lowcode/solution-resources.md` | | Debug / run a low-code agent end-to-end and inspect its output | Low-code | [lowcode/debug.md](references/lowcode/debug.md) | `lowcode/project-lifecycle.md` § `uip agent debug` | | Embed coded agent in a flow (solution-level) | Coded | [coded/embedding-in-flows.md](references/coded/embedding-in-flows.md) | | diff --git a/skills/uipath-agents/references/lowcode/evaluations/evaluate.md b/skills/uipath-agents/references/lowcode/evaluations/evaluate.md index c53f73501b..2f55cc54d5 100644 --- a/skills/uipath-agents/references/lowcode/evaluations/evaluate.md +++ b/skills/uipath-agents/references/lowcode/evaluations/evaluate.md @@ -31,7 +31,7 @@ Local operations (managing evaluators, eval sets, test cases) do **not** require - [Evaluators](evaluators.md) — evaluator types, adding/removing, default prompts - [Evaluation Sets and Test Cases](evaluation-sets.md) — creating sets, adding test cases, simulation options - [Running Evaluations](running-evaluations.md) — start, status, results, compare -- [Orchestrator Runtime Eval Commands](orchestrator-eval-run.md) — full CRUD for evaluators, eval sets, data points, run/schedule/results against published Orchestrator packages +- [Runtime Eval Commands](orchestrator-eval-run.md) — full CRUD for evaluators, eval sets, data points, run/schedule/results via `uip eval` Read Evaluators before choosing an evaluator type, and Evaluation Sets before writing test cases. diff --git a/skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md b/skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md index 1b996b32d1..e69de29bb2 100644 --- a/skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md +++ b/skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md @@ -1,464 +0,0 @@ -# Orchestrator Runtime Eval Commands - -Manage and run runtime evaluations for agents published as Orchestrator packages. All commands are scoped by `--process-key` (process key GUID) and hit the agents runtime API. - -## Command Structure - -``` -uip or eval -├── execute-and-evaluate Submit a runtime eval run -├── run list List eval set runs for a process -├── run get Get details of a specific run -├── run results View per-item results -├── evaluator list/get/create/update/delete Manage evaluators -├── eval-set list/get/create/update/delete Manage eval sets (dataset containers) -├── evaluation list/get/create/update/delete Manage data points within eval sets -└── schedule create/list/get/update/pause/resume/delete - Manage scheduled recurring eval runs -``` - ---- - -## execute-and-evaluate - -Submit a runtime eval run for a published Orchestrator package. - -```bash -uip or eval execute-and-evaluate \ - --process-key \ - --workload-id \ - --items \ - --evaluators \ - [--eval-set-id ] \ - [--batch-size ] \ - [--folder-key ] \ - [--tenant ] \ - --output json -``` - -| Flag | Required | Description | -|------|----------|-------------| -| `--process-key` | Yes | Process key (GUID). Use `uip or processes list` to find keys. | -| `--workload-id` | Yes | Workload ID (GUID). | -| `--items` | Yes | JSON array of eval items. See [Evaluations](#evaluation-data-points). | -| `--evaluators` | Yes | JSON array of evaluator configs. See [Evaluators](#evaluators). | -| `--eval-set-id` | No | Eval set ID; defaults to zero GUID | -| `--batch-size` | No | Max concurrent evaluation pipelines (default: `5`) | -| `--folder-key` | No | Folder key GUID; defaults to personal workspace | -| `--tenant` | No | UiPath tenant name | - -The folder resolves from your personal workspace automatically. Pass `--folder-key` to target a specific folder. - -### Example - -```bash -uip or eval execute-and-evaluate \ - --process-key "9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09" \ - --workload-id "a1b2c3d4-0000-0000-0000-000000000001" \ - --items '[{"id":"i1","name":"Test","inputs":{"input":"hello"},"expectedOutput":{},"expectedBehavior":""}]' \ - --evaluators '[{"id":"ev-1","version":"","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{"name":"Semantic","prompt":"Score 0-100...","model":"gpt-4.1-2025-04-14","targetOutputKey":"*"}}]' \ - --output json -``` - -### Output - -```json -{ - "Result": "Success", - "Code": "EvalRunSubmitted", - "Data": { - "ProcessKey": "9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09", - "Folder": "user@uipath.com's workspace", - "EvalSetId": "00000000-0000-0000-0000-000000000000", - "EvalSetRunId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04" - } -} -``` - ---- - -## Evaluators - -CRUD for evaluators scoped by process key. - -### evaluator list - -```bash -uip or eval evaluator list --process-key [--limit ] [--offset ] [--tenant ] --output json -``` - -Output code: `EvaluatorList`. Fields: EvaluatorId, Name, Description, EvaluatorTypeId, Version, CreatedAt. Includes `Pagination` field. - -### evaluator get - -```bash -uip or eval evaluator get --process-key [--tenant ] --output json -``` - -Output code: `EvaluatorDetails`. - -### evaluator create - -```bash -uip or eval evaluator create \ - --process-key \ - --workload-id \ - --folder-key \ - --name \ - --description \ - --evaluator-type-id \ - --evaluator-config \ - [--version ] \ - [--tenant ] \ - --output json -``` - -| Flag | Required | Description | -|------|----------|-------------| -| `--process-key` | Yes | Process key (GUID) | -| `--workload-id` | Yes | Workload ID (GUID) | -| `--folder-key` | Yes | Folder key (GUID) | -| `--name` | Yes | Evaluator name | -| `--description` | Yes | Evaluator description | -| `--evaluator-type-id` | Yes | Type ID (e.g. `uipath-exact-match`, `uipath-llm-judge-output-semantic-similarity`, `uipath-llm-judge-trajectory-similarity`) | -| `--evaluator-config` | Yes | Evaluator config as JSON object | -| `--version` | No | Version string (default: `1.0`) | - -Output code: `EvaluatorCreated`. - -### evaluator update - -```bash -uip or eval evaluator update \ - --process-key \ - [--name ] \ - [--description ] \ - [--evaluator-type-id ] \ - [--evaluator-config ] \ - [--version ] \ - [--tenant ] \ - --output json -``` - -At least one optional field must be provided. The command fetches the current state, merges your changes, and PUTs the full object back (the backend has no PATCH endpoint). - -Output code: `EvaluatorUpdated`. - -### evaluator delete - -```bash -uip or eval evaluator delete --process-key [--tenant ] --output json -``` - -Output code: `EvaluatorDeleted`. - ---- - -## Eval Sets - -CRUD for eval sets (dataset containers) scoped by process key. - -### eval-set list - -```bash -uip or eval eval-set list --process-key [--limit ] [--offset ] [--tenant ] --output json -``` - -Output code: `EvalSetList`. Fields: EvalSetId, Name, Description, BatchSize, EvaluatorRefs, CreatedAt. Includes `Pagination` field. - -### eval-set get - -```bash -uip or eval eval-set get --process-key [--tenant ] --output json -``` - -Output code: `EvalSetDetails`. - -### eval-set create - -```bash -uip or eval eval-set create \ - --process-key \ - --workload-id \ - --folder-key \ - --name \ - [--description ] \ - [--batch-size ] \ - [--timeout-minutes ] \ - [--evaluator-refs ] \ - [--tenant ] \ - --output json -``` - -| Flag | Required | Description | -|------|----------|-------------| -| `--process-key` | Yes | Process key (GUID) | -| `--workload-id` | Yes | Workload ID (GUID) | -| `--folder-key` | Yes | Folder key (GUID) | -| `--name` | Yes | Eval set name | -| `--description` | No | Description | -| `--batch-size` | No | Max concurrent evaluations | -| `--timeout-minutes` | No | Timeout per evaluation | -| `--evaluator-refs` | No | Evaluator IDs to link (space-separated) | - -Output code: `EvalSetCreated`. - -### eval-set update - -```bash -uip or eval eval-set update \ - --process-key \ - [--name ] \ - [--description ] \ - [--batch-size ] \ - [--timeout-minutes ] \ - [--evaluator-refs ] \ - [--tenant ] \ - --output json -``` - -At least one optional field must be provided. Fetches current state, merges changes, PUTs the full object. - -Output code: `EvalSetUpdated`. - -### eval-set delete - -```bash -uip or eval eval-set delete --process-key [--tenant ] --output json -``` - -Output code: `EvalSetDeleted`. - ---- - -## Evaluation (Data Points) - -CRUD for evaluations (test cases / data points) within eval sets. - -### evaluation list - -```bash -uip or eval evaluation list \ - --process-key \ - --eval-set-id \ - [--limit ] \ - [--offset ] \ - [--tenant ] \ - --output json -``` - -Output code: `EvaluationList`. Fields: EvaluationId, EvalSetId, Name, Inputs, ExpectedOutput, ExpectedBehavior, CreatedAt. Includes `Pagination` field. - -### evaluation get - -```bash -uip or eval evaluation get \ - --process-key \ - --eval-set-id \ - [--tenant ] \ - --output json -``` - -Output code: `EvaluationDetails`. - -### evaluation create - -```bash -uip or eval evaluation create \ - --process-key \ - --eval-set-id \ - --folder-key \ - --name \ - --inputs \ - [--expected-output ] \ - [--expected-behavior ] \ - [--evaluation-criterias ] \ - [--tenant ] \ - --output json -``` - -| Flag | Required | Description | -|------|----------|-------------| -| `--process-key` | Yes | Process key (GUID) | -| `--eval-set-id` | Yes | Eval set ID (GUID) | -| `--folder-key` | Yes | Folder key (GUID) | -| `--name` | Yes | Data point name | -| `--inputs` | Yes | Input values as JSON | -| `--expected-output` | No | Expected output as JSON (for output evaluators) | -| `--expected-behavior` | No | Expected agent behavior (for trajectory evaluators) | -| `--evaluation-criterias` | No | Per-evaluator criteria overrides as JSON (spelling matches the backend API field name) | - -Output code: `EvaluationCreated`. - -### evaluation update - -```bash -uip or eval evaluation update \ - --process-key \ - --eval-set-id \ - [--name ] \ - [--inputs ] \ - [--expected-output ] \ - [--expected-behavior ] \ - [--evaluation-criterias ] \ - [--tenant ] \ - --output json -``` - -At least one optional field must be provided. Fetches current state, merges changes, PUTs the full object. - -Output code: `EvaluationUpdated`. - -### evaluation delete - -```bash -uip or eval evaluation delete \ - --process-key \ - --eval-set-id \ - [--tenant ] \ - --output json -``` - -Output code: `EvaluationDeleted`. - ---- - -## Run Results - -Query eval run results by process key. - -### run list - -```bash -uip or eval run list --process-key [--limit ] [--offset ] [--tenant ] --output json -``` - -Output code: `EvalSetRunList`. Fields: EvalSetRunId, EvalSetId, Status, Score, EvalsExecuted, Duration, CreatedAt. Includes `Pagination` field. - -### run get - -```bash -uip or eval run get --process-key [--tenant ] --output json -``` - -Output code: `EvalSetRunDetails`. - -### run results - -```bash -uip or eval run results --process-key [--tenant ] --output json -``` - -Output code: `EvalRunResults`. Fields: EvalRunId, DataPoint, Status, Result, CreatedAt. - ---- - -## Schedules - -CRUD for scheduled recurring eval runs. - -### schedule create - -```bash -uip or eval schedule create \ - --process-key \ - --eval-set-id \ - --cron \ - [--workload-id ] \ - [--folder-key ] \ - [--tenant ] \ - --output json -``` - -`--process-key`, `--eval-set-id`, and `--cron` are required. `--workload-id` and `--folder-key` are auto-resolved from the eval set when omitted. Pass them explicitly to override. - -Output code: `EvalScheduleCreated`. Fields: ScheduleId, WorkloadId, ProcessKey, FolderKey, EvalSetId, CronExpression, Status, CreatedAt. - -### schedule list / get / update / pause / resume / delete - -```bash -uip or eval schedule list --process-key --output json -uip or eval schedule get --process-key --output json -uip or eval schedule update --process-key [--eval-set-id ] [--cron ] --output json -uip or eval schedule pause --process-key --output json -uip or eval schedule resume --process-key --output json -uip or eval schedule delete --process-key --output json -``` - -Output codes: `EvalScheduleList`, `EvalScheduleDetails`, `EvalScheduleUpdated`, `EvalSchedulePaused`, `EvalScheduleResumed`, `EvalScheduleDeleted`. - -Update requires at least one of `--eval-set-id` or `--cron`. Folder key is immutable after creation. - ---- - -## Typical Workflow — CRUD-first - -Create evaluators, eval sets, and data points via CRUD, then run against the eval set. - -```bash -# 1. Create an evaluator -uip or eval evaluator create \ - --process-key "$PROCESS_KEY" --workload-id "$WORKLOAD_ID" --folder-key "$FOLDER_KEY" \ - --name "Semantic Similarity" --description "LLM output comparison" \ - --evaluator-type-id uipath-llm-judge-output-semantic-similarity \ - --evaluator-config '{"name":"Semantic","prompt":"As an expert evaluator, analyze the semantic similarity of these outputs to determine a score from 0-100.\n----\nExpectedOutput:\n{{ExpectedOutput}}\n----\nActualOutput:\n{{ActualOutput}}\n","model":"gpt-4.1-2025-04-14","targetOutputKey":"*"}' \ - --output json - -# 2. Create an eval set linking the evaluator -uip or eval eval-set create \ - --process-key "$PROCESS_KEY" --workload-id "$WORKLOAD_ID" --folder-key "$FOLDER_KEY" \ - --name "Smoke Tests" --evaluator-refs "$EVALUATOR_ID" \ - --output json - -# 3. Add data points to the eval set -uip or eval evaluation create \ - --process-key "$PROCESS_KEY" --eval-set-id "$EVAL_SET_ID" --folder-key "$FOLDER_KEY" \ - --name "Greeting test" --inputs '{"input":"hello"}' \ - --expected-output '{"content":"Hi there!"}' \ - --output json - -# 4. Update the eval set to add more evaluator refs if needed -uip or eval eval-set update "$EVAL_SET_ID" \ - --process-key "$PROCESS_KEY" \ - --evaluator-refs "$EVALUATOR_ID" "$ANOTHER_EVALUATOR_ID" \ - --output json - -# 5. Run the eval — items and evaluators are passed inline -uip or eval execute-and-evaluate \ - --process-key "$PROCESS_KEY" \ - --workload-id "$WORKLOAD_ID" \ - --eval-set-id "$EVAL_SET_ID" \ - --items '[{"id":"i1","name":"Greeting test","inputs":{"input":"hello"},"expectedOutput":{"content":"Hi there!"},"expectedBehavior":""}]' \ - --evaluators '[{"id":"'"$EVALUATOR_ID"'","version":"","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{"name":"Semantic","prompt":"Score 0-100...","model":"gpt-4.1-2025-04-14","targetOutputKey":"*"}}]' \ - --output json - -# 6. Check results -uip or eval run list --process-key "$PROCESS_KEY" --output json -uip or eval run results "$EVAL_SET_RUN_ID" --process-key "$PROCESS_KEY" --output json - -# 7. Schedule recurring runs (workload-id and folder-key auto-resolved from eval set) -uip or eval schedule create \ - --process-key "$PROCESS_KEY" \ - --eval-set-id "$EVAL_SET_ID" \ - --cron "0 9 * * *" --output json -``` - -## Troubleshooting - -| Error | Cause | Fix | -|-------|-------|-----| -| `401 Unauthorized` | Auth expired | Run `uip login` | -| `Authentication failed` | No active session | Run `uip login` first | -| `Process not found` | Invalid process key | Verify with `uip or processes list` | -| `personal workspace not found` | No personal workspace | Pass `--folder-key` explicitly | -| `WorkloadId must not be equal to zero GUID` | Missing or zero `--workload-id` | Pass a valid workload ID GUID | -| `--items is not a valid JSON array` | Malformed JSON | Check JSON syntax; must be array of objects | -| `--evaluator-config is not valid JSON` | Malformed JSON | Pass a valid JSON object | - -## Anti-patterns - -- **Don't pass `evaluatorConfig: {}` (empty) in `--evaluators`.** LLM-based evaluators (types 5, 7) need `prompt`, `model`, and `targetOutputKey` in the config. An empty config will fail at runtime. -- **Don't pass `"model": "same-as-agent"` in inline evaluator configs.** Runtime evals have no access to `agent.json` to resolve this. Use an explicit model ID. -- **Don't forget `--folder-key` on create commands when not using the personal workspace.** The default personal workspace fallback only works for `execute-and-evaluate`. CRUD commands (`evaluator create`, `eval-set create`, `evaluation create`) require `--folder-key` explicitly. -- **Keep CRUD data and inline `execute-and-evaluate` items in sync.** `execute-and-evaluate` requires `--items` and `--evaluators` inline even when `--eval-set-id` is provided — the inline data is what actually runs. Use CRUD to manage the canonical dataset and copy items from it into `--items` when triggering a run. Divergence between the two causes confusing results. -- **Don't reuse evaluator IDs across different processes.** Evaluators are scoped to a process key. Using IDs from one process in another will fail. diff --git a/tests/tasks/uipath-agents/lowcode/eval_run/eval_run.yaml b/tests/tasks/uipath-agents/lowcode/eval_run/eval_run.yaml index 48b8da56f1..87921e4701 100644 --- a/tests/tasks/uipath-agents/lowcode/eval_run/eval_run.yaml +++ b/tests/tasks/uipath-agents/lowcode/eval_run/eval_run.yaml @@ -18,7 +18,7 @@ description: > there, hand-writing UUIDs is a documented anti-pattern. Deliberately scoped to a SINGLE eval run. `uip agent eval run compare` needs a second run, and observed cloud-run duration varies ~4x - (393s-1476s). Compare and `uip or eval run-offline-evals` belong in their own + (393s-1476s). Compare and `uip eval run-offline-evals` belong in their own task. tags: [uipath-agents, e2e, mode:operate, lifecycle:setup, low-code, feature:eval] From a9d8e4094c3f8bab50cd048c10f0201aa581ec2d Mon Sep 17 00:00:00 2001 From: Anipik Date: Fri, 4 Sep 2026 10:23:25 -0700 Subject: [PATCH 2/4] docs: update eval commands from `uip or eval` to `uip eval` Eval commands moved from orchestrator-tool to a standalone eval-tool package in UiPath/cli#4025. Commands are now `uip eval ...` instead of `uip or eval ...`. Updates the command catalogue in uip-catalog-snapshot.json: removes the old `or eval` entries and adds `eval` entries in their place. Co-Authored-By: Claude Opus 4.6 (1M context) --- assets/uip-catalog-snapshot.json | 64 ++++++++++++++++---------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/assets/uip-catalog-snapshot.json b/assets/uip-catalog-snapshot.json index 54a89dee5f..3a578545fa 100644 --- a/assets/uip-catalog-snapshot.json +++ b/assets/uip-catalog-snapshot.json @@ -412,6 +412,38 @@ "df records update", "docsai", "docsai ask", + "eval", + "eval eval-set", + "eval eval-set create", + "eval eval-set delete", + "eval eval-set get", + "eval eval-set list", + "eval eval-set update", + "eval evaluation", + "eval evaluation create", + "eval evaluation delete", + "eval evaluation get", + "eval evaluation list", + "eval evaluation update", + "eval evaluator", + "eval evaluator create", + "eval evaluator delete", + "eval evaluator get", + "eval evaluator list", + "eval evaluator update", + "eval execute-and-evaluate", + "eval run", + "eval run get", + "eval run list", + "eval run results", + "eval schedule", + "eval schedule create", + "eval schedule delete", + "eval schedule get", + "eval schedule list", + "eval schedule pause", + "eval schedule resume", + "eval schedule update", "feedback", "feedback send", "function", @@ -996,38 +1028,6 @@ "or credential-stores", "or credential-stores get", "or credential-stores list", - "or eval", - "or eval eval-set", - "or eval eval-set create", - "or eval eval-set delete", - "or eval eval-set get", - "or eval eval-set list", - "or eval eval-set update", - "or eval evaluation", - "or eval evaluation create", - "or eval evaluation delete", - "or eval evaluation get", - "or eval evaluation list", - "or eval evaluation update", - "or eval evaluator", - "or eval evaluator create", - "or eval evaluator delete", - "or eval evaluator get", - "or eval evaluator list", - "or eval evaluator update", - "or eval execute-and-evaluate", - "or eval run", - "or eval run get", - "or eval run list", - "or eval run results", - "or eval schedule", - "or eval schedule create", - "or eval schedule delete", - "or eval schedule get", - "or eval schedule list", - "or eval schedule pause", - "or eval schedule resume", - "or eval schedule update", "or feeds", "or feeds list", "or folders", From 6293a62928169ffcfe7cb19b3ee543b8cd7f93bd Mon Sep 17 00:00:00 2001 From: Anipik Date: Fri, 4 Sep 2026 10:30:27 -0700 Subject: [PATCH 3/4] fix: restore orchestrator-eval-run.md content with uip eval prefix The previous commit emptied the file instead of updating it in place. Restores all 464 lines with `uip or eval` replaced by `uip eval`. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../evaluations/orchestrator-eval-run.md | 464 ++++++++++++++++++ 1 file changed, 464 insertions(+) diff --git a/skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md b/skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md index e69de29bb2..dd0e924e99 100644 --- a/skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md +++ b/skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md @@ -0,0 +1,464 @@ +# Orchestrator Runtime Eval Commands + +Manage and run runtime evaluations for agents published as Orchestrator packages. All commands are scoped by `--process-key` (process key GUID) and hit the agents runtime API. + +## Command Structure + +``` +uip eval +├── execute-and-evaluate Submit a runtime eval run +├── run list List eval set runs for a process +├── run get Get details of a specific run +├── run results View per-item results +├── evaluator list/get/create/update/delete Manage evaluators +├── eval-set list/get/create/update/delete Manage eval sets (dataset containers) +├── evaluation list/get/create/update/delete Manage data points within eval sets +└── schedule create/list/get/update/pause/resume/delete + Manage scheduled recurring eval runs +``` + +--- + +## execute-and-evaluate + +Submit a runtime eval run for a published Orchestrator package. + +```bash +uip eval execute-and-evaluate \ + --process-key \ + --workload-id \ + --items \ + --evaluators \ + [--eval-set-id ] \ + [--batch-size ] \ + [--folder-key ] \ + [--tenant ] \ + --output json +``` + +| Flag | Required | Description | +|------|----------|-------------| +| `--process-key` | Yes | Process key (GUID). Use `uip or processes list` to find keys. | +| `--workload-id` | Yes | Workload ID (GUID). | +| `--items` | Yes | JSON array of eval items. See [Evaluations](#evaluation-data-points). | +| `--evaluators` | Yes | JSON array of evaluator configs. See [Evaluators](#evaluators). | +| `--eval-set-id` | No | Eval set ID; defaults to zero GUID | +| `--batch-size` | No | Max concurrent evaluation pipelines (default: `5`) | +| `--folder-key` | No | Folder key GUID; defaults to personal workspace | +| `--tenant` | No | UiPath tenant name | + +The folder resolves from your personal workspace automatically. Pass `--folder-key` to target a specific folder. + +### Example + +```bash +uip eval execute-and-evaluate \ + --process-key "9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09" \ + --workload-id "a1b2c3d4-0000-0000-0000-000000000001" \ + --items '[{"id":"i1","name":"Test","inputs":{"input":"hello"},"expectedOutput":{},"expectedBehavior":""}]' \ + --evaluators '[{"id":"ev-1","version":"","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{"name":"Semantic","prompt":"Score 0-100...","model":"gpt-4.1-2025-04-14","targetOutputKey":"*"}}]' \ + --output json +``` + +### Output + +```json +{ + "Result": "Success", + "Code": "EvalRunSubmitted", + "Data": { + "ProcessKey": "9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09", + "Folder": "user@uipath.com's workspace", + "EvalSetId": "00000000-0000-0000-0000-000000000000", + "EvalSetRunId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04" + } +} +``` + +--- + +## Evaluators + +CRUD for evaluators scoped by process key. + +### evaluator list + +```bash +uip eval evaluator list --process-key [--limit ] [--offset ] [--tenant ] --output json +``` + +Output code: `EvaluatorList`. Fields: EvaluatorId, Name, Description, EvaluatorTypeId, Version, CreatedAt. Includes `Pagination` field. + +### evaluator get + +```bash +uip eval evaluator get --process-key [--tenant ] --output json +``` + +Output code: `EvaluatorDetails`. + +### evaluator create + +```bash +uip eval evaluator create \ + --process-key \ + --workload-id \ + --folder-key \ + --name \ + --description \ + --evaluator-type-id \ + --evaluator-config \ + [--version ] \ + [--tenant ] \ + --output json +``` + +| Flag | Required | Description | +|------|----------|-------------| +| `--process-key` | Yes | Process key (GUID) | +| `--workload-id` | Yes | Workload ID (GUID) | +| `--folder-key` | Yes | Folder key (GUID) | +| `--name` | Yes | Evaluator name | +| `--description` | Yes | Evaluator description | +| `--evaluator-type-id` | Yes | Type ID (e.g. `uipath-exact-match`, `uipath-llm-judge-output-semantic-similarity`, `uipath-llm-judge-trajectory-similarity`) | +| `--evaluator-config` | Yes | Evaluator config as JSON object | +| `--version` | No | Version string (default: `1.0`) | + +Output code: `EvaluatorCreated`. + +### evaluator update + +```bash +uip eval evaluator update \ + --process-key \ + [--name ] \ + [--description ] \ + [--evaluator-type-id ] \ + [--evaluator-config ] \ + [--version ] \ + [--tenant ] \ + --output json +``` + +At least one optional field must be provided. The command fetches the current state, merges your changes, and PUTs the full object back (the backend has no PATCH endpoint). + +Output code: `EvaluatorUpdated`. + +### evaluator delete + +```bash +uip eval evaluator delete --process-key [--tenant ] --output json +``` + +Output code: `EvaluatorDeleted`. + +--- + +## Eval Sets + +CRUD for eval sets (dataset containers) scoped by process key. + +### eval-set list + +```bash +uip eval eval-set list --process-key [--limit ] [--offset ] [--tenant ] --output json +``` + +Output code: `EvalSetList`. Fields: EvalSetId, Name, Description, BatchSize, EvaluatorRefs, CreatedAt. Includes `Pagination` field. + +### eval-set get + +```bash +uip eval eval-set get --process-key [--tenant ] --output json +``` + +Output code: `EvalSetDetails`. + +### eval-set create + +```bash +uip eval eval-set create \ + --process-key \ + --workload-id \ + --folder-key \ + --name \ + [--description ] \ + [--batch-size ] \ + [--timeout-minutes ] \ + [--evaluator-refs ] \ + [--tenant ] \ + --output json +``` + +| Flag | Required | Description | +|------|----------|-------------| +| `--process-key` | Yes | Process key (GUID) | +| `--workload-id` | Yes | Workload ID (GUID) | +| `--folder-key` | Yes | Folder key (GUID) | +| `--name` | Yes | Eval set name | +| `--description` | No | Description | +| `--batch-size` | No | Max concurrent evaluations | +| `--timeout-minutes` | No | Timeout per evaluation | +| `--evaluator-refs` | No | Evaluator IDs to link (space-separated) | + +Output code: `EvalSetCreated`. + +### eval-set update + +```bash +uip eval eval-set update \ + --process-key \ + [--name ] \ + [--description ] \ + [--batch-size ] \ + [--timeout-minutes ] \ + [--evaluator-refs ] \ + [--tenant ] \ + --output json +``` + +At least one optional field must be provided. Fetches current state, merges changes, PUTs the full object. + +Output code: `EvalSetUpdated`. + +### eval-set delete + +```bash +uip eval eval-set delete --process-key [--tenant ] --output json +``` + +Output code: `EvalSetDeleted`. + +--- + +## Evaluation (Data Points) + +CRUD for evaluations (test cases / data points) within eval sets. + +### evaluation list + +```bash +uip eval evaluation list \ + --process-key \ + --eval-set-id \ + [--limit ] \ + [--offset ] \ + [--tenant ] \ + --output json +``` + +Output code: `EvaluationList`. Fields: EvaluationId, EvalSetId, Name, Inputs, ExpectedOutput, ExpectedBehavior, CreatedAt. Includes `Pagination` field. + +### evaluation get + +```bash +uip eval evaluation get \ + --process-key \ + --eval-set-id \ + [--tenant ] \ + --output json +``` + +Output code: `EvaluationDetails`. + +### evaluation create + +```bash +uip eval evaluation create \ + --process-key \ + --eval-set-id \ + --folder-key \ + --name \ + --inputs \ + [--expected-output ] \ + [--expected-behavior ] \ + [--evaluation-criterias ] \ + [--tenant ] \ + --output json +``` + +| Flag | Required | Description | +|------|----------|-------------| +| `--process-key` | Yes | Process key (GUID) | +| `--eval-set-id` | Yes | Eval set ID (GUID) | +| `--folder-key` | Yes | Folder key (GUID) | +| `--name` | Yes | Data point name | +| `--inputs` | Yes | Input values as JSON | +| `--expected-output` | No | Expected output as JSON (for output evaluators) | +| `--expected-behavior` | No | Expected agent behavior (for trajectory evaluators) | +| `--evaluation-criterias` | No | Per-evaluator criteria overrides as JSON (spelling matches the backend API field name) | + +Output code: `EvaluationCreated`. + +### evaluation update + +```bash +uip eval evaluation update \ + --process-key \ + --eval-set-id \ + [--name ] \ + [--inputs ] \ + [--expected-output ] \ + [--expected-behavior ] \ + [--evaluation-criterias ] \ + [--tenant ] \ + --output json +``` + +At least one optional field must be provided. Fetches current state, merges changes, PUTs the full object. + +Output code: `EvaluationUpdated`. + +### evaluation delete + +```bash +uip eval evaluation delete \ + --process-key \ + --eval-set-id \ + [--tenant ] \ + --output json +``` + +Output code: `EvaluationDeleted`. + +--- + +## Run Results + +Query eval run results by process key. + +### run list + +```bash +uip eval run list --process-key [--limit ] [--offset ] [--tenant ] --output json +``` + +Output code: `EvalSetRunList`. Fields: EvalSetRunId, EvalSetId, Status, Score, EvalsExecuted, Duration, CreatedAt. Includes `Pagination` field. + +### run get + +```bash +uip eval run get --process-key [--tenant ] --output json +``` + +Output code: `EvalSetRunDetails`. + +### run results + +```bash +uip eval run results --process-key [--tenant ] --output json +``` + +Output code: `EvalRunResults`. Fields: EvalRunId, DataPoint, Status, Result, CreatedAt. + +--- + +## Schedules + +CRUD for scheduled recurring eval runs. + +### schedule create + +```bash +uip eval schedule create \ + --process-key \ + --eval-set-id \ + --cron \ + [--workload-id ] \ + [--folder-key ] \ + [--tenant ] \ + --output json +``` + +`--process-key`, `--eval-set-id`, and `--cron` are required. `--workload-id` and `--folder-key` are auto-resolved from the eval set when omitted. Pass them explicitly to override. + +Output code: `EvalScheduleCreated`. Fields: ScheduleId, WorkloadId, ProcessKey, FolderKey, EvalSetId, CronExpression, Status, CreatedAt. + +### schedule list / get / update / pause / resume / delete + +```bash +uip eval schedule list --process-key --output json +uip eval schedule get --process-key --output json +uip eval schedule update --process-key [--eval-set-id ] [--cron ] --output json +uip eval schedule pause --process-key --output json +uip eval schedule resume --process-key --output json +uip eval schedule delete --process-key --output json +``` + +Output codes: `EvalScheduleList`, `EvalScheduleDetails`, `EvalScheduleUpdated`, `EvalSchedulePaused`, `EvalScheduleResumed`, `EvalScheduleDeleted`. + +Update requires at least one of `--eval-set-id` or `--cron`. Folder key is immutable after creation. + +--- + +## Typical Workflow — CRUD-first + +Create evaluators, eval sets, and data points via CRUD, then run against the eval set. + +```bash +# 1. Create an evaluator +uip eval evaluator create \ + --process-key "$PROCESS_KEY" --workload-id "$WORKLOAD_ID" --folder-key "$FOLDER_KEY" \ + --name "Semantic Similarity" --description "LLM output comparison" \ + --evaluator-type-id uipath-llm-judge-output-semantic-similarity \ + --evaluator-config '{"name":"Semantic","prompt":"As an expert evaluator, analyze the semantic similarity of these outputs to determine a score from 0-100.\n----\nExpectedOutput:\n{{ExpectedOutput}}\n----\nActualOutput:\n{{ActualOutput}}\n","model":"gpt-4.1-2025-04-14","targetOutputKey":"*"}' \ + --output json + +# 2. Create an eval set linking the evaluator +uip eval eval-set create \ + --process-key "$PROCESS_KEY" --workload-id "$WORKLOAD_ID" --folder-key "$FOLDER_KEY" \ + --name "Smoke Tests" --evaluator-refs "$EVALUATOR_ID" \ + --output json + +# 3. Add data points to the eval set +uip eval evaluation create \ + --process-key "$PROCESS_KEY" --eval-set-id "$EVAL_SET_ID" --folder-key "$FOLDER_KEY" \ + --name "Greeting test" --inputs '{"input":"hello"}' \ + --expected-output '{"content":"Hi there!"}' \ + --output json + +# 4. Update the eval set to add more evaluator refs if needed +uip eval eval-set update "$EVAL_SET_ID" \ + --process-key "$PROCESS_KEY" \ + --evaluator-refs "$EVALUATOR_ID" "$ANOTHER_EVALUATOR_ID" \ + --output json + +# 5. Run the eval — items and evaluators are passed inline +uip eval execute-and-evaluate \ + --process-key "$PROCESS_KEY" \ + --workload-id "$WORKLOAD_ID" \ + --eval-set-id "$EVAL_SET_ID" \ + --items '[{"id":"i1","name":"Greeting test","inputs":{"input":"hello"},"expectedOutput":{"content":"Hi there!"},"expectedBehavior":""}]' \ + --evaluators '[{"id":"'"$EVALUATOR_ID"'","version":"","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{"name":"Semantic","prompt":"Score 0-100...","model":"gpt-4.1-2025-04-14","targetOutputKey":"*"}}]' \ + --output json + +# 6. Check results +uip eval run list --process-key "$PROCESS_KEY" --output json +uip eval run results "$EVAL_SET_RUN_ID" --process-key "$PROCESS_KEY" --output json + +# 7. Schedule recurring runs (workload-id and folder-key auto-resolved from eval set) +uip eval schedule create \ + --process-key "$PROCESS_KEY" \ + --eval-set-id "$EVAL_SET_ID" \ + --cron "0 9 * * *" --output json +``` + +## Troubleshooting + +| Error | Cause | Fix | +|-------|-------|-----| +| `401 Unauthorized` | Auth expired | Run `uip login` | +| `Authentication failed` | No active session | Run `uip login` first | +| `Process not found` | Invalid process key | Verify with `uip or processes list` | +| `personal workspace not found` | No personal workspace | Pass `--folder-key` explicitly | +| `WorkloadId must not be equal to zero GUID` | Missing or zero `--workload-id` | Pass a valid workload ID GUID | +| `--items is not a valid JSON array` | Malformed JSON | Check JSON syntax; must be array of objects | +| `--evaluator-config is not valid JSON` | Malformed JSON | Pass a valid JSON object | + +## Anti-patterns + +- **Don't pass `evaluatorConfig: {}` (empty) in `--evaluators`.** LLM-based evaluators (types 5, 7) need `prompt`, `model`, and `targetOutputKey` in the config. An empty config will fail at runtime. +- **Don't pass `"model": "same-as-agent"` in inline evaluator configs.** Runtime evals have no access to `agent.json` to resolve this. Use an explicit model ID. +- **Don't forget `--folder-key` on create commands when not using the personal workspace.** The default personal workspace fallback only works for `execute-and-evaluate`. CRUD commands (`evaluator create`, `eval-set create`, `evaluation create`) require `--folder-key` explicitly. +- **Keep CRUD data and inline `execute-and-evaluate` items in sync.** `execute-and-evaluate` requires `--items` and `--evaluators` inline even when `--eval-set-id` is provided — the inline data is what actually runs. Use CRUD to manage the canonical dataset and copy items from it into `--items` when triggering a run. Divergence between the two causes confusing results. +- **Don't reuse evaluator IDs across different processes.** Evaluators are scoped to a process key. Using IDs from one process in another will fail. From ebe9b3a98146afb233310fe58ab5dbaa59e9b161 Mon Sep 17 00:00:00 2001 From: Anipik Date: Tue, 8 Sep 2026 11:19:38 -0700 Subject: [PATCH 4/4] =?UTF-8?q?docs:=20add=20retired=20`or=20eval`=20?= =?UTF-8?q?=E2=86=92=20`eval`=20verb=20to=20cli-renames=20registry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses PR review feedback — ensures /lint-task hints at the canonical `uip eval` spelling when authors use the old `uip or eval` prefix. Co-Authored-By: Claude Opus 4.6 (1M context) --- .claude/rules/cli-renames.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.claude/rules/cli-renames.md b/.claude/rules/cli-renames.md index a6dec42122..2e3cb8db70 100644 --- a/.claude/rules/cli-renames.md +++ b/.claude/rules/cli-renames.md @@ -15,6 +15,7 @@ The catalog at `assets/uip-catalog-snapshot.json` is the source of truth for whi | `functions` | `function` | uip 1.200.0| | `sc` | `aops` | uip 1.201.0| | `cicd` | `aops` | uip 1.201.0| +| `or eval` | `eval` | uip 1.202.0| ## When to add an entry