Skip to content

docs: update eval commands from uip or eval to uip eval - #3092

Open
AAgnihotry wants to merge 4 commits into
mainfrom
refactor/eval-commands-moved-to-eval-tool
Open

docs: update eval commands from uip or eval to uip eval#3092
AAgnihotry wants to merge 4 commits into
mainfrom
refactor/eval-commands-moved-to-eval-tool

Conversation

@AAgnihotry

@AAgnihotry AAgnihotry commented Sep 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Updates all uip or eval references to uip eval across skill docs and test tasks
  • Eval commands moved from orchestrator-tool to a standalone eval-tool package in UiPath/cli#4025

Merge order

This PR must merge after UiPath/cli#4025 and after @uipath/eval-tool is published to the channel the nightly refresh job installs from. Otherwise, tonight's refresh-uip-catalog.yml run will regenerate the snapshot from a CLI without uip eval, reverting the 32 entries back to or eval *.

Changed files

  • skills/uipath-agents/SKILL.md — capability table row
  • skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md — full reference doc (all command examples)
  • skills/uipath-agents/references/lowcode/evaluations/evaluate.md — cross-reference link text
  • tests/tasks/uipath-agents/lowcode/eval_run/eval_run.yaml — test task comment
  • assets/uip-catalog-snapshot.json — hand-added 32 eval * verbs (will be regenerated by nightly refresh after cli#4025 lands)
  • .claude/rules/cli-renames.md — added retired or eval → eval row

Test plan

  • Grep confirms zero remaining uip or eval references

🤖 Generated with Claude Code

@AAgnihotry @claude
docs: update eval commands from uip or eval to uip eval
8c03605 
Eval commands moved from orchestrator-tool to a standalone eval-tool
package in UiPath/cli#4025. Commands are now `uip eval ...` instead
of `uip or eval ...`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Claude finished @AAgnihotry's task in 1m 59s —— View job

PR Review: uip or eval → uip eval rename

  • Read review criteria and project conventions
  • Read full diff and changed files
  • Check for remaining uip or eval references
  • Verify file integrity (emptied file concern)
  • Post review findings

Summary

Renames uip or eval to uip eval in four files following the CLI's move of eval commands from orchestrator-tool to a standalone eval-tool package. The uip or eval string is fully eliminated — grep confirms zero remaining references.

Change-by-Change Review

1. skills/uipath-agents/SKILL.md (line 72)

Severity: OK
Capability table cell updated from uip or eval execute-and-evaluate to uip eval execute-and-evaluate. Correct and minimal.

2. skills/uipath-agents/references/lowcode/evaluations/evaluate.md (line 34)

Severity: OK
Link text updated from "Orchestrator Runtime Eval Commands" to "Runtime Eval Commands", and description updated to reference uip eval. The link target (orchestrator-eval-run.md) still exists on disk and is still referenced from SKILL.md — consistent.

3. skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md

Severity: Critical — file emptied, not updated
This file was emptied to 0 bytes (464 lines removed, 0 added). The SHA e69de29bb2d is the git empty blob — this is a completely empty file, not a deleted file. However:

  • SKILL.md:72 still links to it: [lowcode/evaluations/orchestrator-eval-run.md](references/lowcode/evaluations/orchestrator-eval-run.md)
  • evaluate.md:34 still links to it: [Runtime Eval Commands](orchestrator-eval-run.md)

Both links now resolve to an empty file. An agent following either link will get zero content — the entire eval CRUD reference (evaluators, eval sets, data points, runs, schedules, troubleshooting, anti-patterns) is gone with no replacement.

This is the main issue with this PR. The file should have been either:

  • (a) Updated in place — replace every uip or eval with uip eval in the 464-line body (the stated intent of the PR), or
  • (b) Deleted with its links removed from SKILL.md and evaluate.md (if the content is moving elsewhere).

Currently it's neither — it's an empty file that two documents still point to.

Fix this →

4. tests/tasks/uipath-agents/lowcode/eval_run/eval_run.yaml (line 21)

Severity: OK
Comment-only change in the description field: uip or eval run-offline-evals → uip eval run-offline-evals. Correct and does not affect test behavior.

What's Missing

  • The 464-line reference body. The file orchestrator-eval-run.md contained the complete CRUD reference for runtime eval commands (evaluators, eval sets, data points, run results, schedules, troubleshooting, anti-patterns). It was emptied instead of updated. The content either needs to be restored with the new uip eval prefix, or the file and its inbound links need to be removed.

Area Ratings

Area Status Notes
Frontmatter OK Not modified
E2E Tests OK Test task comment updated correctly
Skill Body OK SKILL.md table cell updated correctly
References & Assets Issue orchestrator-eval-run.md emptied — two dangling links to an empty file
Repo Hygiene OK No secrets, scoped changes, no unrelated modifications

Issues for Manual Review

  • Confirm whether the orchestrator-eval-run.md content is intentionally being removed (moved to another location or deprecated) or was accidentally emptied. The PR description says "Updates all uip or eval references to uip eval" which implies a rename, not a content removal.

Conclusion

Request changes. The rename in SKILL.md, evaluate.md, and the test YAML is correct and clean. However, orchestrator-eval-run.md was emptied to 0 bytes instead of being updated — leaving two links pointing to an empty file and removing the entire runtime eval CRUD reference. This needs to be resolved before merging: either restore the content with uip eval substituted, or delete the file and clean up the links.

@github-actions

github-actions Bot commented Sep 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Claude finished @AAgnihotry's task in 1m 42s —— View job

Coder-eval task lint (advisory)

1 task YAML changed; 0 Critical, 0 High, 0 Medium, 0 Low, 1 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

❌ High — PR body does not claim the changed task has been run and passed. The change is documentation-only (one word in the description field, line 21), so re-running is very low risk — but please edit the PR description to add a line like:

Ran skill-agents-lowcode-eval-run locally and it passed.

Or, since this is a comment-only change that cannot affect execution, a note like "Change is description-only; no behavioral impact" would also satisfy this.

Per-task lint

tests/tasks/uipath-agents/lowcode/eval_run/eval_run.yaml — verdict: OK

The diff touches only the description field (line 21: uip or eval → uip eval). No structural, prompt, or criteria changes. The task itself is well-constructed:

  • No self-report anti-pattern — criteria grade CLI side-effects and a check script, not agent prose
  • Prompt states the goal without prescribing procedure
  • Strong meaningful coverage — mix of command_executed, command_not_executed, and a weighted run_command check script
  • Not gameable — criteria require actual uip agent eval CLI invocations
  • No near-duplicates (only task in its folder; unique eval coverage per description)
  • run_limits correctly at top level
  • No redundant @uipath/cli in sandbox

Within-PR duplicates

No duplicate clusters detected (only 1 task YAML changed).

Conclusion

✅ All changed tasks pass the rubric. Evidence of passing run not confirmed — flagged above (advisory).

@AAgnihotry @claude
docs: update eval commands from uip or eval to uip eval
a9d8e40 
Eval commands moved from orchestrator-tool to a standalone eval-tool
package in UiPath/cli#4025. Commands are now `uip eval ...` instead
of `uip or eval ...`.

Updates the command catalogue in uip-catalog-snapshot.json: removes
the old `or eval` entries and adds `eval` entries in their place.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@AAgnihotry @claude
fix: restore orchestrator-eval-run.md content with uip eval prefix
6293a62 
The previous commit emptied the file instead of updating it in place.
Restores all 464 lines with `uip or eval` replaced by `uip eval`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
rockymadden
rockymadden reviewed Sep 4, 2026
View reviewed changes

@rockymadden rockymadden left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against a local build of cli#4025 rather than just reading the diff. The doc edits are correct, mechanical, and complete. My concerns are about the generated file and cross-repo timing, not the rename itself.

Verified, not assumed

Check Result
Content integrity orchestrator-eval-run.md is 464 lines on both main and head; SKILL.md 87/87. Nothing lost — worth confirming since one commit is titled "restore orchestrator-eval-run.md content"
Leftover or eval refs Zero across skills/, preview/, tests/
Snapshot invariants Still sorted, still unique, 32 in / 32 out
Snapshot accuracy I walked the real command tree off the built eval-tool from cli#4025 — the 32 entries match exactly, no extras, nothing missing
CI 54 pass, 3 skipped. Verb-gate re-ran across every skill (a snapshot edit triggers a full re-gate) and passed

Major — uip-catalog-snapshot.json is a generated file, and this creates a merge-order hazard

That file is produced nightly by .github/workflows/refresh-uip-catalog.yml, which installs a live uip and walks it. Its header records provenance:

"generated_at": "2026-09-04T04:08:47Z",
"cli_version": "1.202.0-dev.8521",

Hand-adding the 32 eval * verbs makes the file assert that build ships uip eval. It doesn't, until cli#4025 merges and @uipath/eval-tool is published to the channel the refresh job installs from.

Failure mode if this merges first: tonight's refresh regenerates from a CLI with no uip eval, reverts the 32 entries back to or eval *, and the verb-gate then fails skills/uipath-agents — whose docs now say uip eval. Self-inflicted red on an unrelated auto-PR.

Fix: merge cli#4025 first and confirm the eval-tool is published, then merge this. Either let the nightly refresh regenerate the snapshot on its own, or keep the hand-edit but land both in the right order. Either way, state the ordering constraint in the PR body so whoever hits merge knows.

Minor — missing .claude/rules/cli-renames.md row

That file is the documented registry of retired verbs and feeds /lint-task's Medium CLI-verb hint via scripts/check-cli-verbs.py. This rename qualifies:

+| `or eval`              | `eval`               | uip 1.202.0|

The doc's "drop the row once grep returns nothing" clause doesn't argue against adding it — I checked, and uip functions and uip resource both have 0 remaining references in tests/ and skills/ while their rows are still in the table. Without the row, an author or agent writing uip or eval ... (the spelling in every pre-1.202 doc, and in model training data) gets no replacement hint.

Minor — PR body omits the snapshot from "Changed files"

The list names 4 of the 5 changed files, and the omitted one is the generated file carrying the cross-repo timing constraint. That's precisely the file a reviewer needs pointed at.

Nit — orchestrator-eval-run.md filename is now half-stale

The commands no longer live in the orchestrator tool. The content is still legitimately Orchestrator-scoped (process key = an Orchestrator release), so the SKILL.md row label reads fine, but the filename now implies the tool prefix is or. Two inbound links to update (SKILL.md, evaluate.md) if you rename. Fine to leave it and accept the drift.

Nit — the one test-plan box is unchecked

- [ ] Grep confirms zero remaining uip or eval references is the trivially verifiable item, and it does pass. Confirmed.

Not blocking on the diff — it's accurate and green. The one change I'd ask for is the cli-renames.md row; the rest is sequencing, which is a merge-button decision. Add the row, note the ordering, merge after cli#4025, and this is good to go.

@AAgnihotry @claude
docs: add retired or eval → eval verb to cli-renames registry
ebe9b3a 
Addresses PR review feedback — ensures /lint-task hints at the canonical
`uip eval` spelling when authors use the old `uip or eval` prefix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
rockymadden
rockymadden approved these changes Sep 10, 2026
View reviewed changes

@rockymadden rockymadden left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review. Verified against the live CLI (uip 1.203.0 installed, @uipath/eval-tool@1.203.0-dev.8620 published) and a local merge into current main, not just the diff.

What

uip or eval → uip eval across uipath-agents docs, one test YAML, the catalog snapshot, and the rename registry. Follows the CLI's move of eval commands out of orchestrator-tool into a standalone eval-tool package (UiPath/cli#4025).

My last-pass asks

Ask Status
Add the or eval → eval row to cli-renames.md ✅ ebe9b3a
State the merge-order constraint in the PR body ✅ "Merge order" section
List the snapshot under "Changed files" ✅ added
Merge after cli#4025 + eval-tool publish ✅ moot, both landed

Verified, not assumed

Check Result
uip or eval actually retired? Yes. uip or eval --help → unknown command 'eval'; no eval under uip or
Doc command tree Matches uip eval --help exactly, all 7 groups
Doc flags Spot-checked execute-and-evaluate, evaluator create, schedule create against 1.203.0. Every documented flag exists, no drift
run-offline-evals in the test YAML Real alias of execute-and-evaluate, confirmed in its usage line
Leftover refs Zero uip or eval in the tree post-merge
Coverage Only 4 files in the repo mention these commands. All 4 updated
Links 6779 relative links resolve
Verb gate, uipath-agents main 90 soft-stale / 0 blocking → merged 53 / 0. Clears 37 warnings

The timing hazard I flagged resolved itself

The nightly refresh already regenerated main's snapshot:

snapshot cli_version eval * or eval *
2026-09-09 1.203.0-dev.8572 0 32
2026-09-10 1.203.0-dev.8609 32 0

So main today ships docs naming a verb its own catalog no longer has. This PR is the fix, not the risk.

Minor — the snapshot hunk is now dead weight

The branch is 86 commits behind. Its snapshot is 1.202.0-dev.8521, 18 verbs short of main (agent review-history, insights groups, insights roles, login profiles, rpa remote start|status, ...).

I ran the merge locally: git resolves it cleanly and the snapshot comes out unchanged from main. Nothing regresses. The 32-line hunk just contributes nothing at this point. A rebase drops it; leaving it is harmless.

Recommendation

Approve. Two nits inline, neither worth a round trip. Main is wrong today and this fixes it.

tl;dr — Rename is correct and complete, checked against the real 1.203.0 CLI rather than the diff. The cross-repo timing hazard from my last pass is gone: cli#4025 landed and the nightly refresh already put uip eval in main's snapshot. The snapshot hunk here is redundant but merges to a no-op. Approve, two nits.

Comment thread .claude/rules/cli-renames.md
| `functions` | `function` | uip 1.200.0|
| `sc` | `aops` | uip 1.201.0|
| `cicd` | `aops` | uip 1.201.0|
| `or eval` | `eval` | uip 1.202.0|

@rockymadden rockymadden Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit — retired-at version is off by one minor.

Snapshot history puts the cut inside the 1.203.0-dev line:

  • 1.203.0-dev.8572 (2026-09-09): 32 or eval *, 0 eval *
  • 1.203.0-dev.8609 (2026-09-10): 0 or eval *, 32 eval *
-| `or eval`              | `eval`               | uip 1.202.0|
+| `or eval`              | `eval`               | uip 1.203.0|

This file states column 3 is informational only, so nothing breaks. It is still the column someone reads to date the break.

Comment thread skills/uipath-agents/references/lowcode/evaluations/evaluate.md
- [Evaluation Sets and Test Cases](evaluation-sets.md) — creating sets, adding test cases, simulation options
- [Running Evaluations](running-evaluations.md) — start, status, results, compare
- [Orchestrator Runtime Eval Commands](orchestrator-eval-run.md) — full CRUD for evaluators, eval sets, data points, run/schedule/results against published Orchestrator packages
- [Runtime Eval Commands](orchestrator-eval-run.md) — full CRUD for evaluators, eval sets, data points, run/schedule/results via `uip eval`

@rockymadden rockymadden Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit — this link text now disagrees with the target's H1.

orchestrator-eval-run.md:1 is still # Orchestrator Runtime Eval Commands, but the link here dropped "Orchestrator". Either spelling is defensible (the content is genuinely Orchestrator-scoped: process key = a release), just not one each.

Either restore it here:

-- [Runtime Eval Commands](orchestrator-eval-run.md)
++ [Orchestrator Runtime Eval Commands](orchestrator-eval-run.md)

or drop it from the H1 in the target file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rockymadden rockymadden rockymadden approved these changes
@andreitava-uip andreitava-uip andreitava-uip approved these changes
@AlexandruCGhimisi AlexandruCGhimisi Awaiting requested review from AlexandruCGhimisi AlexandruCGhimisi is a code owner
@andreibalas-uipath andreibalas-uipath Awaiting requested review from andreibalas-uipath andreibalas-uipath is a code owner
@al3xanndru al3xanndru Awaiting requested review from al3xanndru al3xanndru is a code owner
@gabrielavaduva gabrielavaduva Awaiting requested review from gabrielavaduva gabrielavaduva is a code owner
@AlvinStanescu AlvinStanescu Awaiting requested review from AlvinStanescu AlvinStanescu is a code owner
@marius-bughiu marius-bughiu Awaiting requested review from marius-bughiu marius-bughiu is a code owner
@RaduAna-Maria RaduAna-Maria Awaiting requested review from RaduAna-Maria RaduAna-Maria is a code owner
@uipreliga uipreliga Awaiting requested review from uipreliga uipreliga is a code owner
@dmetzgar dmetzgar Awaiting requested review from dmetzgar dmetzgar is a code owner
@gozhang2 gozhang2 Awaiting requested review from gozhang2 gozhang2 is a code owner
@smflorentino smflorentino Awaiting requested review from smflorentino smflorentino is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AAgnihotry @rockymadden @andreitava-uip