feat(desktop): Computer Use for macOS, Windows, and Linux - #6537
feat(desktop): Computer Use for macOS, Windows, and Linux#6537sheehanmunim wants to merge 81 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
vouch:unvouched
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
There was a problem hiding this comment.
One finding: the new desktop-control resolver hides its ServerSettingsService dependency behind Effect.serviceOption, so the requirement is absent from the type of every adapter that calls it and a missing layer silently resolves to "Computer Use enabled". Everything else in the changed TypeScript (subpath namespace imports for effect/*, Effect.fn helpers acquiring FileSystem/Path/HostProcess* from the environment, new Schema.is predicates that add real logic, static CodexAppServerRequestError.methodNotFound factory, no new ManagedRuntime/runPromise/fake Layer.succeed) matches the conventions.
Posted via Macroscope — Effect Service Conventions
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Adds cross-platform “Computer Use” to T3 Code Desktop by bundling a local desktop-control MCP server (Swift on macOS, Rust on Windows/Linux) and an agent-owned Chrome tab workflow via a bundled extension, plus end-to-end settings/approvals plumbing across contracts, server adapters, desktop IPC, and the web UI.
Changes:
- Bundle and resolve a
t3-desktopMCP server across macOS/Windows/Linux, staging it into desktop build artifacts and injecting it into supported provider adapters when enabled. - Add Chrome extension bridge + install scripts, plus a new Settings → Computer Use page that surfaces enablement, extension status, and macOS privacy permission readiness.
- Extend contracts and runtime ingestion to support new approval kinds (
tool/permissions) and Codex MCP form-elicitation flows, with accompanying test coverage.
Reviewed changes
Copilot reviewed 60 out of 75 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| vite.config.ts | Adjust staged formatter globs to avoid native/ files breaking commits |
| scripts/build-desktop-artifact.ts | Stage desktop MCP binaries (Swift/Rust) and macOS Info.plist Apple Events usage text |
| scripts/build-desktop-artifact.test.ts | Add tests for MCP executable naming/targets and macOS usage description |
| packages/effect-codex-app-server/src/errors.ts | Extend identifier purpose literals for new approval request types |
| packages/contracts/src/settings.ts | Add desktopControl settings subtree + patch schema |
| packages/contracts/src/providerRuntime.ts | Add canonical request types for tool/permissions approvals |
| packages/contracts/src/orchestration.ts | Extend ProviderRequestKind to include tool/permissions |
| packages/contracts/src/ipc.ts | Add IPC schemas + DesktopBridge methods for Computer Use readiness |
| package.json | Add build script for Swift desktop MCP server |
| native/t3-desktop-mcp/Sources/BrowserBridge.swift | Swift Chrome-extension bridge via unix socket + native messaging framing |
| native/t3-desktop-mcp/Package.swift | SwiftPM package definition for macOS desktop MCP server |
| native/t3-desktop-mcp/.gitignore | Ignore SwiftPM build output |
| native/t3-desktop-mcp-rs/src/tools.rs | Rust tool schema definitions matching macOS tool surface (with browser_* gating) |
| native/t3-desktop-mcp-rs/src/platform/mod.rs | Cross-platform desktop backend trait + shared formatting/types |
| native/t3-desktop-mcp-rs/src/main.rs | Rust MCP server: newline-delimited JSON-RPC over stdio + tool dispatch |
| native/t3-desktop-mcp-rs/src/capture.rs | Shared display/window capture with Wayland/X11 fallbacks and PNG encoding |
| native/t3-desktop-mcp-rs/src/browser.rs | Rust Chrome bridge + native-host relay over local socket |
| native/t3-desktop-mcp-rs/src/apps.rs | App/window enumeration and pid resolution shared across backends |
| native/t3-desktop-mcp-rs/Cargo.toml | New Rust crate definition + deps for Windows/Linux automation/capture |
| native/t3-desktop-mcp-rs/.gitignore | Ignore Rust target output |
| native/t3-chrome-extension/manifest.json | MV3 extension manifest (native messaging + debugger/tabGroups perms) |
| native/t3-chrome-extension/install.sh | macOS/Linux host registration script for native messaging |
| native/t3-chrome-extension/install.ps1 | Windows registry-based host registration script |
| native/t3-chrome-extension/background.js | Chrome service worker implementing agent-owned tab group + DevTools-based automation |
| apps/web/src/session-logic.ts | Teach pending approval derivation about tool/permissions kinds |
| apps/web/src/session-logic.test.ts | Tests for deriving/retaining new approval kinds in UI work log |
| apps/web/src/routeTree.gen.ts | Add generated route for Settings → Computer Use |
| apps/web/src/routes/settings.computer-use.tsx | Route entry for the Computer Use settings page |
| apps/web/src/components/settings/SettingsSidebarNav.tsx | Add sidebar icon mapping for Computer Use section |
| apps/web/src/components/settings/settingsSearch.ts | Add search items/labels for Computer Use settings and permissions |
| apps/web/src/components/settings/ComputerUseSettings.tsx | New settings UI: toggles, extension status, macOS permission readiness + deep links |
| apps/web/src/components/settings/browserBrandIcons.tsx | Brand icons for browser/agent cursor display in settings |
| apps/web/src/components/chat/ComposerPendingApprovalPanel.tsx | UI labels for tool vs permission pending approvals |
| apps/web/src/components/chat/ComposerPendingApprovalPanel.test.tsx | Tests for new pending-approval labels |
| apps/web/src/assets/computer-use/google-chrome.svg | Add Chrome brand asset |
| apps/web/src/assets/computer-use/firefox.svg | Add Firefox brand asset (not supported yet) |
| apps/web/src/assets/computer-use/edge.svg | Add Edge brand asset |
| apps/web/src/assets/computer-use/brave.svg | Add Brave brand asset |
| apps/server/src/provider/Layers/GrokAdapter.ts | Inject resolved desktop MCP server into Grok adapter MCP server list |
| apps/server/src/provider/Layers/CursorAdapter.ts | Inject resolved desktop MCP server into Cursor adapter MCP server list |
| apps/server/src/provider/Layers/CodexSessionRuntime.ts | Add permissions approval handling + MCP elicitation (tool/permissions) request plumbing |
| apps/server/src/provider/Layers/CodexSessionRuntime.test.ts | Unit tests for permission approvals + MCP elicitation mapping/response building |
| apps/server/src/provider/Layers/CodexProvider.ts | Advertise Codex MCP form-elicitation capability |
| apps/server/src/provider/Layers/CodexProvider.test.ts | Test for Codex initialize capability flags |
| apps/server/src/provider/Layers/CodexAdapter.ts | Map new request kinds/types and include elicitation detail extraction |
| apps/server/src/provider/Layers/CodexAdapter.test.ts | Lifecycle tests for tool/permissions canonical approval mapping |
| apps/server/src/provider/Layers/ClaudeAdapter.ts | Add desktop MCP server into Claude mcpServers map when resolvable |
| apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.ts | Preserve new approval kinds through activity summaries/payload |
| apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.approval.test.ts | Tests ensuring new approval kinds reach clients correctly |
| apps/server/src/desktopControl/resolveEnabledDesktopMcp.ts | Resolve desktop MCP only when enabled; pass env flags for cursor/browser toggles |
| apps/server/src/desktopControl/resolveEnabledDesktopMcp.test.ts | Tests for enable/disable and env flag behavior |
| apps/server/src/desktopControl/desktopMcpBinary.ts | Locate bundled/override MCP binary across platforms and dev layouts |
| apps/server/src/desktopControl/desktopMcpBinary.test.ts | Tests for override resolution and unsupported platform behavior |
| apps/desktop/src/preload.ts | Expose Computer Use IPC methods to renderer via preload bridge |
| apps/desktop/src/ipc/methods/computerUse.ts | IPC handlers for permissions readiness + opening privacy panes |
| apps/desktop/src/ipc/DesktopIpcHandlers.ts | Register new Computer Use IPC handlers |
| apps/desktop/src/ipc/channels.ts | Add IPC channel constants for Computer Use |
| apps/desktop/src/computerUse/permissions.ts | Read macOS TCC statuses + detect Chrome extension/host registration |
| apps/desktop/src/computerUse/permissions.test.ts | Tests for permission status mapping, Settings deep links, and extension detection |
| apps/desktop/scripts/electron-launcher.mjs | Patch macOS Info.plist to include Apple Events usage description |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
ApprovabilityVerdict: Needs human review 11 blocking correctness issues found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR. You can customize Macroscope's approvability policy. Learn more. |
Babysit updateBranch is merge-clean onto current Review fixes pushed (settings fail-closed + per-session resolve without widening session Note for reviewers / Macroscope: the Effect Service Conventions check keeps reporting the old Fork CI workflows may still need org approval; Vercel marketing fails with “Authorization required to deploy” (expected for fork PRs). |
|
Follow-up correctness gaps from review are now fixed on tip:
Windows/Linux desktop MCP packaging was already wired through `stageDesktopMcpRust` in desktop artifact builds — once this lands in Nightly, hand-copying the binary should no longer be needed. |
Note on Macroscope Effect Service ConventionsThis check is still failing with a retained finding against the deleted path `apps/server/src/desktopControl/resolveEnabledDesktopMcp.ts` (`Effect.serviceOption` / `onNone` defaults). That file was renamed to `desktopMcpLaunch.ts`, and the current implementation acquires settings with `yield* ServerSettings.ServerSettingsService` (fail-closed on `getSettings` errors). There is no `serviceOption` / `onNone` path left in the PR. Correctness Check is green; unresolved review threads are at 0. Remaining merge blockers appear to be human review + fork Vercel auth, not code correctness. |
|
Addressed remaining bot findings on
Vercel marketing deploy auth failure is outside this PR’s scope. |
Match the Computer History Macroscope fixes on Computer Use without pulling in the History daemon entrypoint. Co-authored-by: Cursor <cursoragent@cursor.com>
Tag extension disconnect sentinels with connection generation, preserve AT-SPI ambiguity errors in resolve_pid, and reject non-representable click coordinates on Computer Use. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep under-point clicks on the resolved app PID, reject off-window coordinate fallbacks, and sample browser disconnect generation under the outgoing lock. Co-authored-by: Cursor <cursoragent@cursor.com>
Prefer result tab IDs in select/close confirmations, only constrain under-point clicks when `app` is explicit, and surface not-visible element errors from resolvePoint for right-click and drag. Co-authored-by: Cursor <cursoragent@cursor.com>
Skip T3AgentCursor when resolving the window under a click/drag point, and route coordinate-only drags through that under-point window instead of the last inspected Registry.targetPid. Co-authored-by: Cursor <cursoragent@cursor.com>
Look up args.app via resolveApp instead of resolveTargetPid, and reject any drag whose under-point destination is not the source window. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep cross-window rejection for endpoints outside the source frame, but do not use frontmost-under-point alone — that falsely rejects in-window background drags covered by another app. Co-authored-by: Cursor <cursoragent@cursor.com>
Align Computer Use settings with the Source Control / Provider disclosure pattern so UI Consistency checks see the shared Collapsible primitives. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Guard bridge socket probing and writes by connection generation, keep minimized apps in list_apps, fail closed on partial control files, escape app-list delimiters, drop grim all-outputs display fallback, and tighten Chrome agent-window persistence and screenshot scaling. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Serialize ensureAgentWindow under the state lock and keep the Computer Use permission dialog open when System Settings fails to launch. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep History-only CLI wiring off the mac/computer-use branch. Co-authored-by: Cursor <cursoragent@cursor.com>
…ttings error handling Only unlink bridge sockets this listener owns, stop advertising grim-only displays without capture support, and catch rejected privacy-settings open promises. Co-authored-by: Cursor <cursoragent@cursor.com>
… branch Apply Macroscope fixes for agent cursor lifecycle, browser bridge writes, AX window pairing, app resolution, JSON-RPC parse errors, and Linux cursor init without History-only daemon changes. Co-authored-by: Cursor <cursoragent@cursor.com>
Prevent whitespace-only app arguments from matching every running app via substring fallback. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Also strip accidental computer-history CLI from the Computer Use branch. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep Computer Use History-free and avoid short PID queries binding unrelated apps via substring name matches. Co-authored-by: Cursor <cursoragent@cursor.com>
9cec357 to
ddbbb19
Compare
A leftover closing brace from stripping the History CLI broke the top-level Swift script structure after native-host. Co-authored-by: Cursor <cursoragent@cursor.com>
| query: &str, | ||
| ) -> Result<(ElementRef, String, u32)> { | ||
| let trimmed = query.trim(); | ||
| if let Ok(pid) = trimmed.parse::<u32>() { |
There was a problem hiding this comment.
🟠 High platform/linux.rs:655
A numeric query of "0" returns the first application whose PID lookup failed, so get_app_state or activate_app targets an arbitrary unresolved app instead of rejecting the invalid or ambiguous PID. Because applications() uses PID 0 as its failure sentinel, reject pid == 0 before searching the application list.
+ if pid == 0 {
+ return Err(DesktopError::new("pid 0 is not a valid application pid"));
+ }
if let Some(hit) = applications.iter().find(|(_, _, app_pid)| *app_pid == pid) {🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/linux.rs around line 655:
A numeric query of `"0"` returns the first application whose PID lookup failed, so `get_app_state` or `activate_app` targets an arbitrary unresolved app instead of rejecting the invalid or ambiguous PID. Because `applications()` uses PID `0` as its failure sentinel, reject `pid == 0` before searching the application list.
| lines.push(String::new()); | ||
| lines.push(format!("── window {index}: \"{title}\"")); | ||
| let window_start = element_lines.len(); | ||
| self.walk( |
There was a problem hiding this comment.
🟠 High platform/windows.rs:435
get_app_state can traverse far more than max_elements nodes and stall on large UIA trees. walk checks lines.len(), but describe returns None for unnamed Pane nodes, so those visited nodes do not consume the budget; track visited nodes with a separate counter (as the Linux backend does) and enforce max_elements against it.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/windows.rs around line 435:
`get_app_state` can traverse far more than `max_elements` nodes and stall on large UIA trees. `walk` checks `lines.len()`, but `describe` returns `None` for unnamed `Pane` nodes, so those visited nodes do not consume the budget; track visited nodes with a separate counter (as the Linux backend does) and enforce `max_elements` against it.
| fn keycode_for(&self, keysym: u32) -> Result<Option<(Keycode, bool)>> { | ||
| let (mapping, first) = self.keyboard_mapping()?; | ||
| let per = mapping.keysyms_per_keycode as usize; | ||
| for (index, chunk) in mapping.keysyms.chunks(per).enumerate() { |
There was a problem hiding this comment.
🟠 High platform/linux.rs:389
keycode_for ignores keyboard-layout symbols beyond slots 0 and 1, so type_char reports valid AltGr/group characters as absent and fails with “not on the current keyboard layout” when no unused scratch keycode exists. Extend the lookup and returned modifier information to handle all layout levels, rather than falling back to scratch remapping for characters already available in the active layout.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/linux.rs around line 389:
`keycode_for` ignores keyboard-layout symbols beyond slots 0 and 1, so `type_char` reports valid AltGr/group characters as absent and fails with “not on the current keyboard layout” when no unused scratch keycode exists. Extend the lookup and returned modifier information to handle all layout levels, rather than falling back to scratch remapping for characters already available in the active layout.
|
|
||
| /// Parse an `e12`-style element id into its numeric handle. | ||
| fn element_id(raw: &str) -> Result<u32, DesktopError> { | ||
| raw.trim() |
There was a problem hiding this comment.
🟡 Medium src/main.rs:216
element_id accepts malformed IDs such as ee12 and EeE12 as handle 12, so click/type/scroll/set-value operations can act on a real element instead of returning the advertised invalid-ID error. trim_start_matches(['e', 'E']) removes every consecutive prefix character; remove at most one optional prefix with strip_prefix.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/main.rs around line 216:
`element_id` accepts malformed IDs such as `ee12` and `EeE12` as handle `12`, so click/type/scroll/set-value operations can act on a real element instead of returning the advertised invalid-ID error. `trim_start_matches(['e', 'E'])` removes every consecutive prefix character; remove at most one optional prefix with `strip_prefix`.
| let clickCount = (args["click_count"] as? Int) ?? 1 | ||
| guard clickCount > 0, clickCount <= 3 else { | ||
| return "error: click_count must be an integer between 1 and 3" |
There was a problem hiding this comment.
🟠 High Sources/main.swift:817
A non-Int click_count such as "bad" is treated as 1, so toolClick performs an unintended real click instead of returning the stated validation error. Distinguish an omitted value (default to 1) from a malformed supplied value and reject the latter.
- let clickCount = (args["click_count"] as? Int) ?? 1
- guard clickCount > 0, clickCount <= 3 else {
+ let clickCount: Int
+ if let rawClickCount = args["click_count"] {
+ guard let parsedClickCount = rawClickCount as? Int else {
+ return "error: click_count must be an integer between 1 and 3"
+ }
+ clickCount = parsedClickCount
+ } else {
+ clickCount = 1
+ }
+ guard clickCount > 0, clickCount <= 3 else {🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp/Sources/main.swift around lines 817-819:
A non-`Int` `click_count` such as `"bad"` is treated as `1`, so `toolClick` performs an unintended real click instead of returning the stated validation error. Distinguish an omitted value (default to `1`) from a malformed supplied value and reject the latter.
| let is_frontmost = focused_pid | ||
| .filter(|front| *front != 0 && *front == pid) | ||
| .is_some() | ||
| || (!focused_name.is_empty() && name.to_lowercase() == focused_name); |
There was a problem hiding this comment.
🟡 Medium platform/linux.rs:849
list_apps marks every same-named application as FRONTMOST when the focused app and entry both have usable, nonmatching PIDs, so multiple instances produce contradictory frontmost results. The name fallback must only run when either PID is unavailable.
.filter(|front| *front != 0 && *front == pid)
.is_some()
- || (!focused_name.is_empty() && name.to_lowercase() == focused_name);
+ || (pid == 0
+ || focused_pid.is_none_or(|front| front == 0))
+ && !focused_name.is_empty()
+ && name.to_lowercase() == focused_name;🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/linux.rs around lines 849-852:
`list_apps` marks every same-named application as `FRONTMOST` when the focused app and entry both have usable, nonmatching PIDs, so multiple instances produce contradictory frontmost results. The name fallback must only run when either PID is unavailable.
Nonblocking bridge writes, owner-only agent-cursor sockets, no WAYLAND_DISPLAY mutation, Wayland type_text/X11 cookie checks, create-first bridge binding, and inert browser when control is disabled. Co-authored-by: Cursor <cursoragent@cursor.com>
|
|
||
| // Clicks pass through: empty ShapeInput region. Without Shape, a topmost | ||
| // override-redirect window would eat clicks under the hotspot — fail closed. | ||
| if conn.shape_query_version().is_err() |
There was a problem hiding this comment.
🟠 High platform/agent_cursor_linux.rs:403
The overlay is still initialized and mapped when the X server rejects the Shape setup, so its default input region can intercept user clicks instead of passing them through. shape_query_version() and shape_rectangles() only enqueue requests here; their cookies are never checked with reply()/check_for_error(), and the later poll_for_event() consumes the server error after initialization has already proceeded. Validate both request cookies synchronously before setting UI_LIVE or allowing the window to be mapped.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/agent_cursor_linux.rs around line 403:
The overlay is still initialized and mapped when the X server rejects the Shape setup, so its default input region can intercept user clicks instead of passing them through. `shape_query_version()` and `shape_rectangles()` only enqueue requests here; their cookies are never checked with `reply()`/`check_for_error()`, and the later `poll_for_event()` consumes the server error after initialization has already proceeded. Validate both request cookies synchronously before setting `UI_LIVE` or allowing the window to be mapped.
| let ready = poll(&pollFd, 1, waitMs) | ||
| if ready <= 0 { return false } | ||
| continue | ||
| } |
There was a problem hiding this comment.
Nonblocking writes abort socket readers
High Severity
writeAll now sets O_NONBLOCK on the shared bridge socket. serve and the native-host reader treat any read result <= 0 as a hang-up, so EAGAIN during a write tears down the extension connection. A slow or full socket keeps the flag set for the whole deadline, which is exactly when the reader is most likely to wake and exit.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 2fb6531. Configure here.
| connection | ||
| .set_input_focus(InputFocus::PARENT, window_id, 0u32) | ||
| .map_err(|error| DesktopError::new(format!("could not set input focus: {error}")))? | ||
| .check() |
There was a problem hiding this comment.
X11 raise checks fail activation
Medium Severity
raise_x11_window now check()s best-effort restack and set_input_focus nudges. Those requests commonly return BadMatch (unmapped or not-yet-viewable windows) even after a successful _NET_ACTIVE_WINDOW. Activation then errors and never reports success, including cases where the EWMH request already went out.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 2fb6531. Configure here.
…MOST Avoid O_NONBLOCK on the shared socket, revalidate the fd under writeLock, propagate Shift release failures, and stop fabricating FRONTMOST markers. Co-authored-by: Cursor <cursoragent@cursor.com>
| connection | ||
| .xtest_fake_input(if press { 4 } else { 5 }, button, 0, root, 0, 0, 0) | ||
| .map_err(|error| DesktopError::new(format!("could not send button event: {error}")))?; |
There was a problem hiding this comment.
🟠 High platform/linux.rs:368
button returns success after transmitting an XTEST request without checking the X11 protocol result, so rejected button presses are reported as successful and clicks, right-clicks, or scroll events silently do nothing. Chain .check() before flush(), as move_pointer does.
.xtest_fake_input(if press { 4 } else { 5 }, button, 0, root, 0, 0, 0)
- .map_err(|error| DesktopError::new(format!("could not send button event: {error}")))?;
+ .map_err(|error| DesktopError::new(format!("could not send button event: {error}")))?
+ .check()
+ .map_err(|error| DesktopError::new(format!("could not send button event: {error}")))?;🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/linux.rs around lines 368-370:
`button` returns success after transmitting an XTEST request without checking the X11 protocol result, so rejected button presses are reported as successful and clicks, right-clicks, or scroll events silently do nothing. Chain `.check()` before `flush()`, as `move_pointer` does.
Pick the advertised press/click/activate action instead of hard-coding index 0. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 15 total unresolved issues (including 13 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f35e608. Configure here.
| DesktopError::new( | ||
| "element exposes no press/click action — use coordinates or set_value", | ||
| ) | ||
| })?; |
There was a problem hiding this comment.
Matcher uses localized action names
Medium Severity
get_actions() returns localized action names, but the new matcher only compares English press/click/activate. On non-English locales those names do not match, so invoke fails. Native Wayland then has no coordinate fallback, so element clicks stop working after this change.
Reviewed by Cursor Bugbot for commit f35e608. Configure here.
| DesktopError::new( | ||
| "element exposes no press/click action — use coordinates or set_value", | ||
| ) | ||
| })?; |
There was a problem hiding this comment.
Toggle actions are never selected
Medium Severity
The new action picker never treats toggle as a click. Checkboxes, radios, and toggle buttons that only advertise toggle (common in Qt) no longer invoke. That used to work via do_action(0) and now fails on Wayland, where coordinate fallback is blocked.
Reviewed by Cursor Bugbot for commit f35e608. Configure here.
Summary
t3-desktopMCP (Swift on macOS, Rust on Windows/Linux), including agent cursor overlay and background clicks that do not steal the user mouse.origin/main(ahead of current nightlyv0.0.34-nightly.20260813.1087): contracts/desktop/web typecheck clean for this work; Windows Blade E2E confirmedlist_apps, backgroundclick … in background, andscreenshot.Test plan
t3-desktop-mcp.exe,list_appsworks, coordinate click returnsin backgroundlist_apps/ screenshot on X11 (or expected Wayland messaging)native/t3-chrome-extension, verify Settings shows extension installed andbrowser_*tools work in a labelled tab groupmainMade with Cursor
Note
Add Computer Use desktop control for macOS, Windows, and Linux
native/t3-desktop-mcp) and a new Rust MCP server for Windows/Linux (native/t3-desktop-mcp-rs) that expose tools for screen capture, app listing, accessibility-tree inspection, and synthetic input.native/t3-chrome-extension) with a native messaging bridge so the MCP server can control browser tabs; includes install scripts for macOS/Linux and Windows.desktopControlsettings block (enabled,agentCursorEnabled,browserControlEnabled).tool_approval/permissions_approvalrequest types through the Codex session runtime, pending-approvals list, and chat UI./settings/computer-usesettings page with permission status polling, privacy pane links, and browser extension setup guidance.Macroscope summarized f35e608.