fix(uipath-rpa): document both run/debug payload shapes (Helm vs Studio Desktop) and drop the --output-filter mandate

[danmunteanu1]

Why

In a 30-run headless bench of uipath-rpa (5 journeys x 2 skill arms x 3 replicates, @uipath/cli 1.200.1-preview.129), 26 of 30 runs had at least one uip rpa run / debug start rejected with:

Filter '{output: output, errorCount: length(errors), ...}' failed to evaluate:
Invalid type: length() expected argument 1 to be type (string | array | object) but received type null instead.

The filter is the one the skill mandates verbatim in cli-reference.md § Capturing the verdict (Rule 8a says "always pass --output-filter"). Each rejection cost 1-3 retry turns (the workflow had already run, so retries re-drive the application), and agents then spent further turns saving "output-filter null errors" / "run verdict shape" notes to memory. Across the bench this was the largest single source of post-gate turns.

Root cause

The docs described one Data payload — {output: "Session ended", errors: [...], logEntries: [...]} (from #2698) — and derived the verdict rule and the mandated --output-filter from it. That payload is what Studio Desktop returns when the project is open in a running Studio instance. The bench, and every agent running against a project that is not open in Studio, goes through headless Studio (Helm), which returns a different payload: {output, hasErrors, errorMessage, profiling, debugState, debugDetails} with the workflow's log lines streamed above the envelope as [<Level>] … text. Same machine, same project, same CLI, two shapes — selected by the backend, not by the build (reproduced on uip 1.200.1-preview and 1.201.0-preview.133 with rpa-tool 1.200.0-preview, 1.200.1, 1.201.0-preview, 1.202.0-dev).

Consequences on Helm of docs written for the Desktop shape: length(errors) throws because errors does not exist there (26 of 30 bench runs); output is the serialized output arguments, not a status string; the log lines are outside Data; the verdict rule could never be satisfied, so agents fell back to the outer Result, which Rule 8a forbids.

Backend	Observed Data (clean run)	Observed failure states
Helm (no Studio instance has the project open)	{output: "{}", hasErrors: false, errorMessage: null, profiling: null, debugState: null, debugDetails: null}; debug start adds debugState: "Completed"; [Level] log lines above the envelope	faulted / invalid / missing-entry-point run → outer Result: "Failure", Message = the fields JSON-encoded with hasErrors: true; faulted debug start → Suspended, exception in debugDetails; unopenable dir / busy executor → Message: {"success": false, "errorMessage": …}
Studio Desktop (project open in Studio)	{output: "Session ended", errors: [], logEntries: [{source, level, message}], debugState: "Completed"} for run and debug start; nothing streams above the envelope	missing entry point → Result: "Success", {output: "Failed to open the file <path>", errors: [], logEntries: []}; an undeclared --input-arguments key is accepted silently. Unhandled exception: not observed, not documented

Fix (docs only)

• cli-reference.md § Reading run / debug results + § Capturing the verdict: document both payloads keyed by backend (table above), with one verdict rule per shape — Helm: hasErrors == false AND errorMessage == null AND debugState in {null, "Completed"}; Studio Desktop: errors empty AND output == "Session ended" (the missing-entry-point case shows why both are needed). § Headless Studio (Helm) vs Studio Desktop now states the routing that was observed: a running Studio instance that has the project open handles that project's calls, run/debug start included (uip rpa instances list --output json shows who holds what); it previously said Desktop is used only by UI-side-effect verbs. The --output-filter mandate for run / debug start is dropped, not replaced: Data is six short fields (a filter saves nothing), the filter cannot reach the log lines (they stream above the envelope), and any filter naming a key outside Data fails the call after the workflow has already run. One anti-pattern line keeps that failure mode documented. The "never | tail / | head" rule stays with the corrected rationale (log lines precede the envelope). --output-filter remains documented in § Output format for list-shaped responses, where it does help.
• SKILL.md Rule 8a: same verdict fields, output semantics corrected, "read the envelope as printed, no --output-filter, never tail/head".
• debugging.md § Output Format: two subsections, § Helm shape and § Studio Desktop shape, each with its example, field table and verdict callout; § 5 Runtime Validation, § Reading Debug Output Effectively and the profiling notes use the probed camelCase names (profiling.outputDirectory).
• uia-starter-guide.md § Running UI Automation Workflows step 2: debug start without a filter; log lines stream above the envelope.

Removed rather than rewritten, because neither reproduces on either backend: the nested Data.runResult variant, and the "a style diagnostic (IDE0063) can fail the verdict" callout (a coded workflow with a simplifiable using runs with hasErrors: false). The Desktop shape's unhandled-exception state (errors item fields, output: "Execution aborted…") is also not documented — it has not been observed.

Rule numbering, section anchors and every inbound link are unchanged.

Claim-by-claim probe (third commit)

Every factual sentence in the changed sections was re-run against the CLI (uip 1.200.1-preview.129, rpa-tool 1.200.0-preview) with purpose-built fixtures: an entry point with an out argument and an Error-level Log Message, a divide-by-zero Assign, a 25 s Delay, a missing entry point, an unknown flag, an unopenable project dir, --breakpoints, --profiling, debug state in every state. Corrections that came out of it:

• profiling is {"outputDirectory": "..."} (camelCase) — every OutputDirectory mention fixed.
• output is "{}" on a run that faulted inside an activity and "" only when the run never started (validation failure, missing entry point) or while a session is Paused / Suspended / Running; debug continue to completion returns the out-arguments.
• debugDetails is "" (not null) while Running.
• hasErrors causes reduced to the three observed (unhandled exception, compile/validation failure, missing entry point); "cancellation, timeout" removed as unverified.
• errorMessage has three observed forms (activity fault Source:/Message:/Exception Type:; Validation failed with N error(s):; Message: The <file> workflow cannot be found), all listed.
• Outer Failure has two Message shapes: the Data fields JSON-encoded (faulted / invalid / missing-entry-point run) and {"success": false, "errorMessage": ...} (unopenable project dir, busy executor).

Everything else in the sections reproduced as written. Probe script and raw output are in the bench workspace (probe-payload.sh, probe-results.txt, PROBES.md).

Second probe pass (after dropping the filter mandate)

Re-checked the sentences the cleanup introduced: a filter naming an absent key fails only after the workflow ran (the [Error] log line and execution ended print before the ValidationError); log lines and envelope are both on stdout (stderr empty), so > run.log captures everything; Paused debugDetails has exactly Activity, ActivityId, WorkflowFile, CurrentActivity, Locals; the profiling folder holds *.uistat (screenshots appear only for UI activities); profiling is null on a compile failure. Two wording fixes came out of it: log lines are [<Level>] … (not only [Information]), and the Failure envelope's Message has two shapes (Data fields JSON-encoded vs {"success": false, "errorMessage": …}), now stated in cli-reference as well. Example output uses <PROJECT_NAME> and a literal observed Sum: 10 line rather than invented text.

Verification

• Filter and verdict fields probed live on the global install (rpa-tool 1.200.0-preview): clean run, clean debug start, debug start suspended on DivideByZeroException, faulted run. Payload shape re-confirmed on rpa-tool 1.200.1 (latest), 1.201.0-preview.20260828.10 (preview) and 1.202.0-dev.20260825.8 (dev), see the table above.
• node scripts/check-skill-links.mjs: 6602 links resolve. npm run skills:validate: OK (27 skills / 1759 files; studioweb 182 replacements). No skill-flavors/ override touches the four files. No test YAML references the removed strings.

[github-actions]

Claude finished @danmunteanu1's task in 3m 12s —— View job

---

PR Review: fix(uipath-rpa): document the real run/debug payload shape and a null-safe verdict filter

• Read review criteria and project conventions
• Analyze full diff
• Read all changed files in full for context
• Check links, cross-skill refs, secrets
• Post review findings

---

Summary

Corrects the uipath-rpa skill's documentation of the uip rpa run / debug start result payload. The old docs described a {output: "Session ended", errors: [...], logEntries: [...]} shape that no reachable CLI build returns; the actual shape is {output, hasErrors, errorMessage, debugState, debugDetails, profiling}. The copy-paste --output-filter is replaced with a null-safe one (no length()/keys() on absent keys), and verdict logic, examples, and the UIA starter guide are updated to match. Four files changed, all under skills/uipath-rpa/.

Change-by-Change Review

1. skills/uipath-rpa/SKILL.md (+1/−1)

Severity: OK

Rule 8a is updated to reference the verified fields (hasErrors, errorMessage, debugState) and explain the faulted-run vs faulted-debug start distinction. The new text correctly captures the three-field verdict rule, the Data.output semantics correction, the anti-pattern for length()/keys() on absent keys, and the log-line-above-envelope note. Links to cli-reference.md and debugging.md anchors are unchanged and valid.

2. skills/uipath-rpa/references/cli-reference.md (+16/−12)

Severity: OK

• § Reading run / debug results: updated to document the flat hasErrors shape as primary, with runResult as the nested PascalCase variant.
• § Capturing the verdict: the old filter (length(errors)) is replaced with a null-safe filter listing every verdict field explicitly. Example verb changed from debug start to run --skip-build — reasonable since the text says the filter applies to both verbs.
• Example output block changed from json to text fenced block to include the [Information] log lines that stream above the JSON envelope — correctly shows what the agent actually sees.
• Verdict blockquote, the | tail warning, the fallback-file recipe, and the style-diagnostic callout are all updated consistently.
• All links resolve.

3. skills/uipath-rpa/references/debugging.md (+32/−38)

Severity: Low

Core changes are correct and thorough — § Output Format now documents the verified flat shape with a proper field table, the verdict blockquote uses the three-field rule, § Reading Debug Output Effectively is updated, and the § 5 Runtime Validation code comment is corrected.

One item:

• Pre-existing PascalCase in untouched sections (debugging.md:480, debugging.md:303, debugging.md:91): Lines like HasErrors: true and ErrorMessage now sit in a file whose primary § Output Format table uses camelCase hasErrors/errorMessage. These are pre-existing (not introduced by this PR) and can be read as referring to the runResult-nested PascalCase variant documented at line 210. However, the § Identifying the Root Cause example at line 480 doesn't scope itself to the nested form — an agent reading it after § Output Format may be confused about which casing to use. Not introduced by this PR, so not blocking, but worth a follow-up to align the casing or add a brief qualifier (e.g., "The response's hasErrors (or HasErrors in the nested runResult form)").

4. skills/uipath-rpa/references/uia-starter-guide.md (+1/−1)

Severity: OK

Step 2 of the UIA run procedure now points to the copy-paste filter and warns about absent-key failures. The | tail trap parenthetical is replaced with the null-safety rationale. Link to cli-reference.md#capturing-the-verdict resolves. No UIA boundary violations (only references generic uip rpa debug start, not UIA-specific subcommands).

What's Missing

• No residual references to the old shape remain in the changed files — verified via grep. The only "Session ended" reference is the intentional fallback clause at debugging.md:210 for a hypothetical build that emits errors/logEntries keys, which is appropriate.
• No flavor overrides exist for uipath-rpa, so no cross-flavor consistency concern.
• No test YAML references the removed strings ("Session ended", length(errors), Data.logEntries).
• Nothing identified as missing from this PR.

Area Ratings

Area	Status	Notes
Frontmatter	OK	Not changed; still valid
E2E Tests	OK	Docs-only change; existing tests don't reference removed strings
Skill Body	OK	Rule 8a updated consistently
References & Assets	OK	All four reference changes are internally consistent and cross-linked correctly
Repo Hygiene	OK	No secrets, no cross-skill deps, changes scoped to uipath-rpa only

Issues for Manual Review

• CLI payload shape accuracy: The review criteria note that automated reviewers cannot verify whether CLI commands match the tool's actual API. The PR author probed the shape on uip 1.200.1-preview.129 and 1.202.0-dev.8548 — independent confirmation on a GA release would strengthen confidence that the documented shape is stable.
• JMESPath filter null-safety: The claim that logEntries[?level!='Trace'].message | [-8:] evaluates cleanly when logEntries is null is consistent with the JMESPath spec (filter on null → null, slice on null → null), but verifying against the CLI's specific JMESPath implementation would be belt-and-suspenders.

Conclusion

Clean, well-motivated docs fix that corrects a real and measurable agent failure mode (26/30 bench runs hit it). All four files are internally consistent, correctly cross-linked, and free of the old payload shape outside the intentional runResult fallback clause. The only observation is a pre-existing PascalCase inconsistency in untouched sections of debugging.md — not blocking. Approve.

---

[andrei-balint]

I created a jira issue for the rpa-tool side as well https://uipath.atlassian.net/browse/STUD-81410

[AlvinStanescu]

How would the model know it has a running SD instance? It would probably query the processes list, right?

Also, I fear the UIPATH_RPA_TOOL_USE_STUDIO=1 part is outdated. cc @DragosUnguru @RaduAna-Maria

[danmunteanu1]

I ran additional tests with this branch rebased on latest main. The agent never writes about these details, it just runs the command and never comments about the output type, just observes the workflow succeeded.

[danmunteanu1]

It does feel more confident about its output interpretation:

Run passed: hasErrors: false, debugState: "Completed", and the log line reads Calculator result: 5 + 5 = 10.