Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 62 additions & 5 deletions skills/uipath-platform/references/traces/feedback.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Use for agent output quality review and building evaluation datasets.
| `get <id>` | Fetch one feedback record |
| `list` | List feedback with filters |
| `list detailed` | List feedback with span context, plus extra filters (max 200 items) |
| `update <id>` | Change sentiment, comment, or categories |
| `update <id>` | Change sentiment, comment, metadata, or categories |
| `delete <id>` | Remove feedback |

## create
Expand All @@ -32,7 +32,7 @@ uip traces feedback create \
| `--positive` / `--negative` | One required | Mutually exclusive |
| `--folder-key` | Yes | |
| `--span-id` | No | Defaults to root span of trace |
| `--comment` | No | Max 4000 chars; mutually exclusive with `--comment-file` |
| `--comment` | No | Max 1048576 chars; mutually exclusive with `--comment-file` |
| `--comment-file` | No | Path to file; use `-` to read from stdin |
| `--category` | No | Repeatable. Built-in values: `"Output"`, `"Agent Error"`, `"Agent Plan Execution"` |
| `--agent-id` | No | Agent reference GUID |
Expand Down Expand Up @@ -92,7 +92,7 @@ Additional flags over `list`: `--since <duration>`, `--after <ISO>`, `--before <

## update

`--category` tags are **replacement**, not additive — passing `--category` replaces all existing tags.
Positional `<id>`, one of `--positive` / `--negative`, and `--folder-key` required.

```bash
uip traces feedback update <feedback-id> \
Expand All @@ -102,6 +102,59 @@ uip traces feedback update <feedback-id> \
--output json
```

| Flag | Required | Notes |
|------|----------|-------|
| `--positive` / `--negative` | One required | Mutually exclusive |
| `--folder-key` | Yes | |
| `--comment` | No | Max 1048576 chars; mutually exclusive with `--comment-file` |
| `--comment-file` | No | Path to file; use `-` to read from stdin |
| `--metadata` | No | Must be valid JSON. Max 1048576 chars; mutually exclusive with `--metadata-file` |
| `--metadata-file` | No | Path to file; use `-` to read from stdin |
| `--category` | No | Repeatable. **Replacement**, not additive |
| `--profile <name>` | No | Named login profile |

### Omitted fields are preserved

The API replaces the whole record, so the CLI reads it before it writes and carries over every field the caller did not pass. Updating only `--metadata` keeps the existing comment and categories.

Read-modify-write is not atomic: a concurrent edit between the read and the write is lost. The API offers no ETag or PATCH.

### Clearing fields

| Field | Clear with |
|-------|-----------|
| Comment | `--comment ""` |
| Metadata | `--metadata ""` |
| Categories | Not possible — `--category ""` stores a tag literally named `""` |

### Metadata must be valid JSON

Any JSON value is accepted — object, array, string, number. Non-JSON text is rejected server-side with `INVALID_FEEDBACK_METADATA`. The CLI does not pre-validate; the value passes through verbatim. Length is checked before JSON validity.

```bash
uip traces feedback update <feedback-id> \
--positive \
--metadata '{"reviewer":"qa","round":2}' \
--folder-key <folder-key> \
--output json

# From a file (large or nested payloads)
uip traces feedback update <feedback-id> \
--positive \
--metadata-file review.json \
--folder-key <folder-key> \
--output json

# From stdin
jq -n '{reviewer:"qa"}' | uip traces feedback update <feedback-id> \
--positive \
--metadata-file - \
--folder-key <folder-key> \
--output json
```

`create` has no `--metadata` — set metadata with `update` after creating.

## delete

`-y` is required — the CLI never prompts, so a delete without it is rejected.
Expand Down Expand Up @@ -140,8 +193,12 @@ uip traces feedback create \

1. `--positive` / `--negative` — mutually exclusive on all commands
2. `--comment` / `--comment-file` — mutually exclusive on `create` and `update`
3. `--trace-id` — required on `create`; optional filter on `list` / `list detailed`
4. `--folder-key` — required on `create`, `update`, `delete`; optional on `get` / `list`
3. `--metadata` / `--metadata-file` — mutually exclusive on `update`
4. `--comment-file -` / `--metadata-file -` — only one source may read stdin. Both as `-` is rejected: `--comment-file and --metadata-file cannot both read stdin`
5. `--trace-id` — required on `create`; optional filter on `list` / `list detailed`
6. `--folder-key` — required on `create`, `update`, `delete`; optional on `get` / `list`

A flag used against its own `-file` twin is reported before the stdin clash, and both before any file is opened.

## Related

Expand Down
2 changes: 1 addition & 1 deletion skills/uipath-platform/references/uip-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ LLM execution trace observability and feedback annotation. See [traces/traces.md
| `uip traces feedback get <id>` | Fetch one feedback record |
| `uip traces feedback list` | List feedback, optionally filtered by trace/span/agent/sentiment |
| `uip traces feedback list detailed` | Feedback with span context, plus time-range/category/sort filters |
| `uip traces feedback update <id>` | Change sentiment, comment, or categories |
| `uip traces feedback update <id>` | Change sentiment, comment, metadata, or categories |
| `uip traces feedback delete <id>` | Remove feedback |

---
Expand Down
20 changes: 20 additions & 0 deletions tests/tasks/uipath-platform/traces/seed_feedback_metadata.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/bin/sh
# Seeds the structured review payload traces_feedback_metadata_smoke.yaml expects
# in the sandbox: a nested JSON document too large to retype on the command line.
set -eu

cat > review-meta.json <<'JSON'
{
"reviewer": "qa-sweep",
"round": 3,
"rubric": {
"grounding": 2,
"completeness": 4,
"tone": 5
},
"sampledFrom": ["monthly-export", "escalation-queue"],
"notes": "Totals line was dropped on two of the five sampled documents."
}
JSON

echo "OK: seeded review-meta.json"
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
task_id: skill-platform-traces-feedback-metadata
description: >
Smoke test: agent attaches structured review metadata to existing feedback
records via `update --metadata` and `--metadata-file`, neither of which is
reached by any other traces test. Two assertions discriminate. First, metadata
must be valid JSON — the prompt states the payload in prose, so an agent that
passes the phrasing through as bare text is rejected server-side with
INVALID_FEEDBACK_METADATA. Second, `update` now reads the record before it
writes and carries over fields the caller omitted, so re-passing the existing
comment and tags to "protect" them is wrong; the agent must send metadata
alone. Auth/404 responses are acceptable outcomes — the graded behavior is
command shape.
tags: [uipath-platform, smoke, mode:build, lifecycle:setup, traces, feedback]

pre_run:
- command: "sh $SKILLS_REPO_PATH/tests/tasks/uipath-platform/traces/seed_feedback_metadata.sh"
timeout: 15

run_limits:
expected_turns: 10

initial_prompt: |
Two feedback records in folder key `2f6c8a41-93bd-4d17-8e55-6a0b7c19d3e2`
need the review payload from this quarter's sweep attached to them. Both
already carry a reviewer comment and category tags from the original triage,
and none of that may change — only the payload is being added.

- Record `9b41c7e0-5d38-4a62-b1f7-2c8e5a0d94f3`: reviewer is `qa-sweep`, this
is round `3`, and it was sampled from the monthly export. Keep it
thumbs-down, as it is today.
- Record `4e07a2d9-8c15-43f6-9a20-7d3b1f6c8e52`: the payload is the full
rubric already sitting in `review-meta.json`. It is long and nested — do not
retype it. Keep it thumbs-down, as it is today.

Both IDs come from the review export and are authoritative — apply the updates
directly rather than looking the records up first. They may not resolve
against this tenant; that is expected and is not a reason to stop. Report what
each command returns.

success_criteria:
# Segment idiom per tests/README.md: (?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*
# confines every lookahead to THIS command, so a chained second `uip` call
# cannot satisfy or veto an assertion about the first.
# `--metadata[\s=]` (separator required) does not match `--metadata-file`;
# the carry-over criterion spells `--metadata(?:-file)?` so an agent that
# stages even the short payload in a file still earns it.
- type: command_executed
description: "Agent serialized the prose payload as JSON on --metadata, with the required --folder-key"
tool_name: "Bash"
command_pattern: '(uip|\$UIP)\s+traces\s+feedback\s+update\s+9b41c7e0-5d38-4a62-b1f7-2c8e5a0d94f3(?=(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--metadata[\s=]+["'']?\{)(?=(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--folder-key)'
min_count: 1
weight: 2.5
pass_threshold: 1.0

- type: command_executed
description: "Agent sent metadata alone (inline or from a file) — omitted comment and categories are preserved by the read-before-write, so re-passing them is wrong"
tool_name: "Bash"
command_pattern: '(uip|\$UIP)\s+traces\s+feedback\s+update\s+9b41c7e0-5d38-4a62-b1f7-2c8e5a0d94f3(?=(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--metadata(?:-file)?[\s=])(?!(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--comment(?:-file)?[\s=])(?!(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--category[\s=])'
min_count: 1
weight: 3.0
pass_threshold: 1.0

- type: command_executed
description: "Agent sourced the nested rubric from review-meta.json rather than retyping it inline"
tool_name: "Bash"
command_pattern: '(uip|\$UIP)\s+traces\s+feedback\s+update\s+4e07a2d9-8c15-43f6-9a20-7d3b1f6c8e52(?=(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--metadata-file[\s=]+["'']?(?:\$?\S*/)?review-meta\.json)(?!(?:(?!\n|&&|\|\||;|\||\s(?:uip|\$UIP)\s).)*--metadata[\s=])'
min_count: 1
weight: 2.5
pass_threshold: 1.0
Loading