feat(desktop): Computer History for macOS, Windows, and Linux - #1
Closed
sheehanmunim wants to merge 1 commit into
Closed
feat(desktop): Computer History for macOS, Windows, and Linux#1sheehanmunim wants to merge 1 commit into
sheehanmunim wants to merge 1 commit into
Conversation
… Linux Capture accessibility interaction events (not screenshots), summarize them into local memories agents can reference, and optionally mirror Codex skysight. Includes desktop daemon lifecycle, settings UI, provider injection, and Linux FRONTMOST fixes. Co-authored-by: Cursor <cursoragent@cursor.com>
Owner
Author
|
Superseded by upstream pingdotgg#6579 (stacked on pingdotgg#6537). |
There was a problem hiding this comment.
Pull request overview
Adds an opt-in “Computer History” feature across the stack: native desktop daemons record accessibility-driven interaction events into local segments, shared code summarizes segments into markdown “memories”, the server can inject recent history into provider context (Codex), and the web Settings UI controls/inspects the feature via desktop IPC.
Changes:
- Introduces shared Computer History storage/summarization utilities and tests (
packages/shared/src/computerHistory/*). - Adds desktop IPC + native daemons (Swift on macOS; Rust on Windows/Linux) plus a Linux Xvfb smoke harness.
- Wires server-side periodic summarization + Codex developer-instruction/context injection, and adds a web Settings page for controlling/timeline viewing.
Reviewed changes
Copilot reviewed 34 out of 34 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/shared/src/computerHistory/summarize.ts | Generates 10min/6h summaries and renders markdown memory files + instructions text. |
| packages/shared/src/computerHistory/store.ts | Filesystem-backed segment/memory store, summarization pass, timeline listing, mirroring, and clear/delete helpers. |
| packages/shared/src/computerHistory/store.test.ts | Unit tests for filters, summarizer rendering, mirroring, and clear behavior. |
| packages/shared/src/computerHistory/paths.ts | Path helpers for Computer History layout + Codex Skysight mirror destinations. |
| packages/shared/src/computerHistory/index.ts | Public exports for shared Computer History module. |
| packages/shared/src/computerHistory/events.ts | Event/control/status schemas + basic app/website filtering helpers. |
| packages/shared/package.json | Exposes @t3tools/shared/computerHistory export entrypoint. |
| packages/effect-codex-app-server/src/client.ts | Switches request-id tracking from FiberRef to Context.Reference and provides it per-request. |
| packages/contracts/src/settings.ts | Adds computerHistory settings block to server settings schema + patch schema. |
| packages/contracts/src/ipc.ts | Extends desktop bridge interface with Computer History IPC methods/types. |
| packages/contracts/src/index.ts | Exports new Computer History contracts. |
| packages/contracts/src/computerHistory.ts | Adds Computer History contracts (status/timeline/suggestion/clear scope schemas). |
| native/t3-desktop-mcp/Sources/main.swift | Adds computer-history --root entrypoint to macOS desktop MCP binary. |
| native/t3-desktop-mcp/Sources/ComputerHistory.swift | Implements macOS background recorder daemon using NSWorkspace + AX polling. |
| native/t3-desktop-mcp-rs/src/platform/linux.rs | Improves Linux “frontmost” detection/marking for app listing (used by history sampling). |
| native/t3-desktop-mcp-rs/src/main.rs | Adds computer-history --root entrypoint to Rust desktop MCP binary. |
| native/t3-desktop-mcp-rs/src/history.rs | Implements Windows/Linux history daemon: samples frontmost/app-state and writes segments/status. |
| native/t3-desktop-mcp-rs/linux-ch-smoke/run.sh | Adds Linux Xvfb/Openbox-ish smoke script to validate sampling in CI-like env. |
| native/t3-desktop-mcp-rs/linux-ch-smoke/Dockerfile | Container image for the Linux smoke harness. |
| apps/web/src/routeTree.gen.ts | Registers new settings route for Computer History. |
| apps/web/src/routes/settings.computer-history.tsx | Adds the /settings/computer-history route. |
| apps/web/src/components/settings/SettingsSidebarNav.tsx | Adds sidebar nav icon entry for Computer History. |
| apps/web/src/components/settings/settingsSearch.ts | Adds settings search entries/labels for Computer History. |
| apps/web/src/components/settings/ComputerHistorySettings.tsx | Implements Settings UI for toggles, status, and memory timeline actions. |
| apps/server/src/server.ts | Installs Computer History runtime layer into background/server layering. |
| apps/server/src/provider/Layers/CodexSessionRuntime.ts | Injects recent Computer History context into Codex turn start params and updates request-id access. |
| apps/server/src/provider/CodexDeveloperInstructions.ts | Appends optional Computer History context to Codex developer instructions. |
| apps/server/src/computerHistory/service.ts | Adds server helpers for context building, control sync, and periodic summarization loop. |
| apps/desktop/src/preload.ts | Exposes Computer History IPC methods on window.desktopBridge. |
| apps/desktop/src/ipc/methods/computerHistory.ts | Implements IPC handlers (status/timeline/patch/clear/reveal/delete) for desktop. |
| apps/desktop/src/ipc/DesktopIpcHandlers.ts | Registers the new Computer History IPC handlers. |
| apps/desktop/src/ipc/channels.ts | Adds IPC channel constants for Computer History. |
| apps/desktop/src/computerHistory/resolveBinary.ts | Resolves bundled/dev native binary path for daemon spawning. |
| apps/desktop/src/computerHistory/manager.ts | Spawns/stops daemon, syncs control.json, and proxies store operations + reveal. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| return started <= windowStart && !NodeFs.existsSync(NodePath.join(rolledMarkerDir, item.id)); | ||
| }); | ||
|
|
||
| if (candidates.length >= 2) { |
Comment on lines
+1949
to
+1964
| const computerHistoryContext = yield* Effect.tryPromise({ | ||
| try: async () => { | ||
| const { loadRecentContextMarkdown, resolveComputerHistoryRoot } = | ||
| await import("@t3tools/shared/computerHistory"); | ||
| const stateDir = | ||
| process.env.T3CODE_STATE_DIR ?? process.env.T3_STATE_DIR ?? undefined; | ||
| if (!stateDir) return undefined; | ||
| return loadRecentContextMarkdown(resolveComputerHistoryRoot(stateDir)); | ||
| }, | ||
| catch: () => undefined, | ||
| }).pipe( | ||
| Effect.orElseSucceed(() => undefined), | ||
| Effect.map((markdown) => | ||
| markdown ? `<computer_history>\n${markdown}\n</computer_history>` : undefined, | ||
| ), | ||
| ); |
Comment on lines
+41
to
+46
| export const syncComputerHistoryControl = Effect.fn("computerHistory.syncControl")(function* () { | ||
| const config = yield* ServerConfig.ServerConfig; | ||
| const settings = yield* ServerSettings.ServerSettingsService; | ||
| const snapshot = yield* settings.getSettings; | ||
| const history = snapshot.computerHistory; | ||
| const root = resolveComputerHistoryRoot(config.stateDir); |
| return { created: 0 }; | ||
| } | ||
| const root = resolveComputerHistoryRoot(config.stateDir); | ||
| const codexHome = defaultCodexHome(); |
Comment on lines
+38
to
+47
| const refresh = useCallback(async () => { | ||
| const bridge = window.desktopBridge; | ||
| if (!bridge?.getComputerHistoryStatus || !bridge.getComputerHistoryTimeline) return; | ||
| const [nextStatus, timeline] = await Promise.all([ | ||
| bridge.getComputerHistoryStatus(), | ||
| bridge.getComputerHistoryTimeline(), | ||
| ]); | ||
| setStatus(nextStatus); | ||
| setItems([...timeline.items]); | ||
| }, []); |
Comment on lines
+1907
to
+1914
| if CommandLine.arguments.contains("computer-history") { | ||
| let args = CommandLine.arguments | ||
| if let flag = args.firstIndex(of: "--root"), args.index(after: flag) < args.endIndex { | ||
| ComputerHistoryDaemon.run(root: args[args.index(after: flag)]) | ||
| } | ||
| fputs("t3-desktop-mcp: computer-history requires --root <dir>\n", stderr) | ||
| exit(2) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacks on
mac/computer-use/ pingdotgg#6537. Open against upstreammainafter Computer Use lands (or retarget once that branch exists on pingdotgg).Test plan
running; events collected; summaries + Codex mirrort3-desktop-mcp; Xvfb + Openbox smoke recordssample.frontmostfor XTermpackages/shared/src/computerHistory/store.test.ts)Made with Cursor