AI chat panel: build watchfaces and apps from a description - #78
Draft
ericmigi wants to merge 1 commit into
Draft
Conversation
You describe what you want in a panel beside the editor; it writes the code,
builds it on the existing build farm, installs it into the emulator already
running in your browser tab, screenshots the result, and iterates until it looks
right.
Nothing new was built where something already existed. The agent has no
filesystem, no shell and no Pebble SDK: every action is either an HTTPS call to
CloudPebble's own API or a frame on the emulator's phone websocket -- the same
channel the IDE's own install button and emulator buttons already use. The
websocket framing was verified against a real emulator before any of this was
written.
Three pieces:
* The panel (ide/static/ide/js/agent.js) -- a third IDE column, server events
over SSE, no new frontend dependency.
* The server (ide/api/agent.py) -- mints a token scoped to one project, relays
a turn to the agent service, streams events back, stores credentials.
* The agent (cloudpebble-agent/) -- the Claude Agent SDK driving MCP tools that
call back into CloudPebble. Stateless: the transcript lives in the database,
so a turn can be resumed from anywhere.
The scoped token is the security boundary of the feature. It is minted per turn
for a single project, accepted by a hand-picked list of views, and refused for
every sibling project; ide/tests/test_agent_api.py exists mostly to hold that
line. Two pre-existing IDOR holes in the resource views were found and closed
while opening them to the agent.
Users bring their own model -- a Claude subscription, an Anthropic key, or an
OpenRouter key for anything else -- or fall back to a free shared tier. That tier
is a text-only model plus a vision describer rather than a weak vision model: a
model that cannot see is told so and behaves honestly, whereas one handed an
image it cannot read invents a description and edits code against it.
The agent can do what the IDE can: source files across every target (C, pkjs,
Alloy embeddedjs), resources, binary assets, project settings including npm
dependencies, builds, installs, screenshots, logs, and pressing the watch buttons
to drive an app it has written.
Two rounds of end-to-end testing as a user who cannot code, written up in
AGENT_E2E_RUN.md: four projects built entirely through the browser on the free
tier -- two watchfaces, one of them animated, and two apps, one pulling live data
from a public API. Both rounds found real defects and all of them are fixed here.
Deliberately absent: touch input (it reaches the emulator as VNC pointer events,
not over this control channel), and anything that reaches outside the one project
a token is for -- GitHub sync, publishing, interdependencies, deletion, ownership.
Anthropic browser sign-in stays off unless a client id is configured, which it is
not in production.
Off unless AGENT_ENABLED_USERS names someone. Every key is read from the
environment; none is committed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vznsfu5P4sCmohtJCKJU2f
ericmigi
force-pushed
the
claude/agent-chat-panel
branch
from
August 8, 2026 18:49
3e8ee79 to
8651988
Compare
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an AI assistant to the CloudPebble IDE. You describe what you want in the
chat panel; it writes the code, builds it on the existing build farm, installs it
into the emulator already running in your browser tab, screenshots the result and
iterates until it looks right.
Draft: it runs on dev today and has been driven end to end by hand, but it has
not been through review, and there are open items listed at the bottom.
How it fits together
Nothing new was built where something already existed. The agent has no
filesystem, no shell and no Pebble SDK — every action is either an HTTPS call to
CloudPebble's own API or a frame on the emulator's phone websocket, which is the
same channel the IDE's own buttons and installer use.
ide/static/ide/js/agent.js), serverevents over SSE, no new frontend framework.
ide/api/agent.py) — mints a scoped token for one project, relaysa turn to the agent VM, streams events back, and owns the credential storage.
cloudpebble-agent/) — the Claude Agent SDK driving MCP toolsthat call back into CloudPebble. Stateless; the transcript lives in the
database.
The scoped token is the security boundary: it is minted per turn for a single
project, accepted by a hand-picked list of views, and refused for every sibling
project.
ide/tests/test_agent_api.pycovers that boundary specifically.What a user can do
Bring their own model — Claude subscription (browser sign-in or
claude setup-token), an Anthropic key, or an OpenRouter key for anything else. Withoutone they get a free shared tier: a cheap text-only model plus a vision describer,
so it still verifies its own screenshots.
The agent can do what the IDE can: source files across every target (C, pkjs,
Alloy embeddedjs), resources, binary assets, project settings including npm
dependencies, builds, installs, screenshots, logs, and — as of the last commits —
pressing the watch buttons to drive an app it has written.
Evidence
Two rounds of end-to-end testing as a non-coding user, written up in
AGENT_E2E_RUN.md: four projects built through the browser on the free tier —two watchfaces (one animated) and two apps (one pulling live data from a public
API). Both rounds found real defects, all fixed in this branch.
Not included, on purpose
emulator as VNC pointer events rather than over the control channel these
tools use.
ownership transfer — each reaches outside the single project a token is for.
AGENT_ANTHROPIC_OAUTH_CLIENT_IDisset, which it is not in production. See the comment in
ide/api/agent.py.Known open items
overwrite each other's installed app. Pre-existing, but the agent makes it easy
to hit.
Startup now releases them with a message saying "continue" resumes; moving the
relay to celery would remove the interruption.
Config
Off unless
AGENT_ENABLED_USERSnames someone. NeedsAGENT_URLandAGENT_AUTH_HEADERfor the agent service; the free tier needsAGENT_FREE_*and
AGENT_VISION_*. No secrets are committed — every key is read from theenvironment.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Vznsfu5P4sCmohtJCKJU2f