diff --git a/skills/uipath-platform/references/traces/feedback.md b/skills/uipath-platform/references/traces/feedback.md index 01ee7a88a9..92286c6893 100644 --- a/skills/uipath-platform/references/traces/feedback.md +++ b/skills/uipath-platform/references/traces/feedback.md @@ -11,7 +11,7 @@ Use for agent output quality review and building evaluation datasets. | `get ` | Fetch one feedback record | | `list` | List feedback with filters | | `list detailed` | List feedback with span context, plus extra filters (max 200 items) | -| `update ` | Change sentiment, comment, or categories | +| `update ` | Change sentiment, comment, metadata, or categories | | `delete ` | Remove feedback | ## create @@ -32,7 +32,7 @@ uip traces feedback create \ | `--positive` / `--negative` | One required | Mutually exclusive | | `--folder-key` | Yes | | | `--span-id` | No | Defaults to root span of trace | -| `--comment` | No | Max 4000 chars; mutually exclusive with `--comment-file` | +| `--comment` | No | Max 1048576 chars; mutually exclusive with `--comment-file` | | `--comment-file` | No | Path to file; use `-` to read from stdin | | `--category` | No | Repeatable. Built-in values: `"Output"`, `"Agent Error"`, `"Agent Plan Execution"` | | `--agent-id` | No | Agent reference GUID | @@ -92,7 +92,7 @@ Additional flags over `list`: `--since `, `--after `, `--before < ## update -`--category` tags are **replacement**, not additive — passing `--category` replaces all existing tags. +Positional ``, one of `--positive` / `--negative`, and `--folder-key` required. ```bash uip traces feedback update \ @@ -102,6 +102,59 @@ uip traces feedback update \ --output json ``` +| Flag | Required | Notes | +|------|----------|-------| +| `--positive` / `--negative` | One required | Mutually exclusive | +| `--folder-key` | Yes | | +| `--comment` | No | Max 1048576 chars; mutually exclusive with `--comment-file` | +| `--comment-file` | No | Path to file; use `-` to read from stdin | +| `--metadata` | No | Must be valid JSON. Max 1048576 chars; mutually exclusive with `--metadata-file` | +| `--metadata-file` | No | Path to file; use `-` to read from stdin | +| `--category` | No | Repeatable. **Replacement**, not additive | +| `--profile ` | No | Named login profile | + +### Omitted fields are preserved + +The API replaces the whole record, so the CLI reads it before it writes and carries over every field the caller did not pass. Updating only `--metadata` keeps the existing comment and categories. + +Read-modify-write is not atomic: a concurrent edit between the read and the write is lost. The API offers no ETag or PATCH. + +### Clearing fields + +| Field | Clear with | +|-------|-----------| +| Comment | `--comment ""` | +| Metadata | `--metadata ""` | +| Categories | Not possible — `--category ""` stores a tag literally named `""` | + +### Metadata must be valid JSON + +Any JSON value is accepted — object, array, string, number. Non-JSON text is rejected server-side with `INVALID_FEEDBACK_METADATA`. The CLI does not pre-validate; the value passes through verbatim. Length is checked before JSON validity. + +```bash +uip traces feedback update \ + --positive \ + --metadata '{"reviewer":"qa","round":2}' \ + --folder-key \ + --output json + +# From a file (large or nested payloads) +uip traces feedback update \ + --positive \ + --metadata-file review.json \ + --folder-key \ + --output json + +# From stdin +jq -n '{reviewer:"qa"}' | uip traces feedback update \ + --positive \ + --metadata-file - \ + --folder-key \ + --output json +``` + +`create` has no `--metadata` — set metadata with `update` after creating. + ## delete `-y` is required — the CLI never prompts, so a delete without it is rejected. @@ -140,8 +193,12 @@ uip traces feedback create \ 1. `--positive` / `--negative` — mutually exclusive on all commands 2. `--comment` / `--comment-file` — mutually exclusive on `create` and `update` -3. `--trace-id` — required on `create`; optional filter on `list` / `list detailed` -4. `--folder-key` — required on `create`, `update`, `delete`; optional on `get` / `list` +3. `--metadata` / `--metadata-file` — mutually exclusive on `update` +4. `--comment-file -` / `--metadata-file -` — only one source may read stdin. Both as `-` is rejected: `--comment-file and --metadata-file cannot both read stdin` +5. `--trace-id` — required on `create`; optional filter on `list` / `list detailed` +6. `--folder-key` — required on `create`, `update`, `delete`; optional on `get` / `list` + +A flag used against its own `-file` twin is reported before the stdin clash, and both before any file is opened. ## Related diff --git a/skills/uipath-platform/references/uip-commands.md b/skills/uipath-platform/references/uip-commands.md index 44f4cf6f27..d8d597b363 100644 --- a/skills/uipath-platform/references/uip-commands.md +++ b/skills/uipath-platform/references/uip-commands.md @@ -112,7 +112,7 @@ LLM execution trace observability and feedback annotation. See [traces/traces.md | `uip traces feedback get ` | Fetch one feedback record | | `uip traces feedback list` | List feedback, optionally filtered by trace/span/agent/sentiment | | `uip traces feedback list detailed` | Feedback with span context, plus time-range/category/sort filters | -| `uip traces feedback update ` | Change sentiment, comment, or categories | +| `uip traces feedback update ` | Change sentiment, comment, metadata, or categories | | `uip traces feedback delete ` | Remove feedback | --- diff --git a/tests/tasks/uipath-platform/traces/seed_feedback_metadata.sh b/tests/tasks/uipath-platform/traces/seed_feedback_metadata.sh new file mode 100755 index 0000000000..bd8f673a8a --- /dev/null +++ b/tests/tasks/uipath-platform/traces/seed_feedback_metadata.sh @@ -0,0 +1,20 @@ +#!/bin/sh +# Seeds the structured review payload traces_feedback_metadata_smoke.yaml expects +# in the sandbox: a nested JSON document too large to retype on the command line. +set -eu + +cat > review-meta.json <<'JSON' +{ + "reviewer": "qa-sweep", + "round": 3, + "rubric": { + "grounding": 2, + "completeness": 4, + "tone": 5 + }, + "sampledFrom": ["monthly-export", "escalation-queue"], + "notes": "Totals line was dropped on two of the five sampled documents." +} +JSON + +echo "OK: seeded review-meta.json" diff --git a/tests/tasks/uipath-platform/traces/traces_feedback_metadata_smoke.yaml b/tests/tasks/uipath-platform/traces/traces_feedback_metadata_smoke.yaml new file mode 100644 index 0000000000..a54a428313 --- /dev/null +++ b/tests/tasks/uipath-platform/traces/traces_feedback_metadata_smoke.yaml @@ -0,0 +1,69 @@ +task_id: skill-platform-traces-feedback-metadata +description: > + Smoke test: agent attaches structured review metadata to existing feedback + records via `update --metadata` and `--metadata-file`, neither of which is + reached by any other traces test. Two assertions discriminate. First, metadata + must be valid JSON — the prompt states the payload in prose, so an agent that + passes the phrasing through as bare text is rejected server-side with + INVALID_FEEDBACK_METADATA. Second, `update` now reads the record before it + writes and carries over fields the caller omitted, so re-passing the existing + comment and tags to "protect" them is wrong; the agent must send metadata + alone. Auth/404 responses are acceptable outcomes — the graded behavior is + command shape. +tags: [uipath-platform, smoke, mode:build, lifecycle:setup, traces, feedback] + +pre_run: + - command: "sh $SKILLS_REPO_PATH/tests/tasks/uipath-platform/traces/seed_feedback_metadata.sh" + timeout: 15 + +run_limits: + expected_turns: 10 + +initial_prompt: | + Two feedback records in folder key `2f6c8a41-93bd-4d17-8e55-6a0b7c19d3e2` + need the review payload from this quarter's sweep attached to them. Both + already carry a reviewer comment and category tags from the original triage, + and none of that may change — only the payload is being added. + + - Record `9b41c7e0-5d38-4a62-b1f7-2c8e5a0d94f3`: reviewer is `qa-sweep`, this + is round `3`, and it was sampled from the monthly export. Keep it + thumbs-down, as it is today. + - Record `4e07a2d9-8c15-43f6-9a20-7d3b1f6c8e52`: the payload is the full + rubric already sitting in `review-meta.json`. It is long and nested — do not + retype it. Keep it thumbs-down, as it is today. + + Both IDs come from the review export and are authoritative — apply the updates + directly rather than looking the records up first. They may not resolve + against this tenant; that is expected and is not a reason to stop. Report what + each command returns. + +success_criteria: + # Segment idiom per tests/README.md: (?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)* + # confines every lookahead to THIS command, so a chained second `uip` call + # cannot satisfy or veto an assertion about the first. + # `--metadata[\s=]` (separator required) does not match `--metadata-file`; + # the carry-over criterion spells `--metadata(?:-file)?` so an agent that + # stages even the short payload in a file still earns it. + - type: command_executed + description: "Agent serialized the prose payload as JSON on --metadata, with the required --folder-key" + tool_name: "Bash" + command_pattern: '(uip|\$UIP)\s+traces\s+feedback\s+update\s+9b41c7e0-5d38-4a62-b1f7-2c8e5a0d94f3(?=(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--metadata[\s=]+["'']?\{)(?=(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--folder-key)' + min_count: 1 + weight: 2.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent sent metadata alone (inline or from a file) — omitted comment and categories are preserved by the read-before-write, so re-passing them is wrong" + tool_name: "Bash" + command_pattern: '(uip|\$UIP)\s+traces\s+feedback\s+update\s+9b41c7e0-5d38-4a62-b1f7-2c8e5a0d94f3(?=(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--metadata(?:-file)?[\s=])(?!(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--comment(?:-file)?[\s=])(?!(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--category[\s=])' + min_count: 1 + weight: 3.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent sourced the nested rubric from review-meta.json rather than retyping it inline" + tool_name: "Bash" + command_pattern: '(uip|\$UIP)\s+traces\s+feedback\s+update\s+4e07a2d9-8c15-43f6-9a20-7d3b1f6c8e52(?=(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--metadata-file[\s=]+["'']?(?:\$?\S*/)?review-meta\.json)(?!(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--metadata[\s=])' + min_count: 1 + weight: 2.5 + pass_threshold: 1.0