Skip to content

Commit 282ba57

Browse files
committed
Merge main into per-person computers
Brings in a Bot's own browser with Record (#868) and closing browsers when memory runs high (#879). Conflicts are resolved here; the Bot's browser is moved onto per-person seats in the next commit.
2 parents 989970e + 6e28921 commit 282ba57

152 files changed

Lines changed: 16497 additions & 4192 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.example‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -359,6 +359,20 @@ COMPUTER_TOKEN=
359359
# COMPUTER_BROWSER_BACKEND=managed
360360
# COMPUTER_BROWSER_MODE=headed
361361

362+
# When a computer closes a browser besides the cap (COMPUTER_MAX_BROWSERS) and the idle timeout
363+
# (COMPUTER_BROWSER_IDLE_MS). At or above this share of its memory limit, read as the kubelet reads it
364+
# (the working set, so the page cache does not count), it closes browsers nobody has used for a
365+
# minute, least recently used first, until it is ten points under. Memory is read before each launch
366+
# and once a minute, so a page that grows between two checks can still run it out of memory. A
367+
# container with no memory limit is measured against the machine's memory; a computer outside a
368+
# container closes nothing for memory. In the all-in-one image the share counts the database and the
369+
# API server too, on purpose: the out-of-memory kill counts them as well. 0 switches it off.
370+
# COMPUTER_MEMORY_HIGH_PERCENT=85
371+
# How long a Bot's background tab may sit untouched before it is closed, in ms: a tab the Bot left
372+
# behind, such as the page a sign-in popup opened from. The page the Bot is on, a person's own
373+
# browser, and any tab of a Bot whose wheel a person holds are never closed. 0 keeps them.
374+
# COMPUTER_TAB_IDLE_MS=600000
375+
362376
# Web Bot Auth: every request a Bot's browser makes is signed as coming from this deployment
363377
# (Ed25519, HTTP Message Signatures), and the public key is published at
364378
# <OPENBOT_PUBLIC_URL>/.well-known/http-message-signatures-directory. Off unless set to on, because

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -139,13 +139,13 @@ jobs:
139139
# The headless run of the Web Bot Auth test is also the one that checks the user agent fix.
140140
- run: >-
141141
bun test tests/sign-in.test.ts tests/session-cookies.test.ts tests/follows-popup.test.ts
142-
tests/demonstration-live.test.ts tests/webrtc-proxy.test.ts tests/web-bot-auth-browser.test.ts
142+
tests/webrtc-proxy.test.ts tests/web-bot-auth-browser.test.ts
143143
tests/handoff-page-restore.test.ts tests/enter-text.test.ts tests/person-browser.test.ts tests/person-browser-computer.test.ts
144-
tests/screencast-keys-browser.test.ts
144+
tests/bot-browser.test.ts tests/bot-browser-computer.test.ts
145+
tests/screencast-keys-browser.test.ts tests/browser-memory.test.ts
145146
working-directory: agent-computer
146147
env:
147148
OPENBOT_COMPUTER_BROWSER: '1'
148-
OPENBOT_DEMONSTRATION_LIVE: '1'
149149
COMPUTER_BROWSER_MODE: headless
150150
# Headed is the default for hosted computers, so it is tested as one: the computer starting its
151151
# own Xvfb, and the signing again with cross-site frames in processes of their own.

‎CHANGELOG.md‎

Lines changed: 121 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.
88

99
## Unreleased
1010

11-
**Before upgrading.** Fifteen things change for an existing deployment:
11+
**Before upgrading.** Sixteen things change for an existing deployment:
1212
- Bots now share one computer per person instead of each having its own. Migration
1313
`0069_isolated_bots` adds `agent_profiles.isolated`, off for every existing Bot. With the
1414
supervisor (laptop, Desktop, Compose), a person's computer is created the first time one of their
@@ -72,6 +72,55 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.
7272
to keep the old behaviour.
7373
- Interfaces a Bot drew before the upgrade (A2UI 0.9 surfaces) no longer draw when a conversation is
7474
reopened; the messages around them stay. New ones use A2UI 1.0 (below).
75+
- A computer now closes browsers nobody is using, least recently used first, when its memory
76+
reaches 85% of its limit, and closes a Bot's background tabs after ten minutes untouched (below).
77+
In the all-in-one image that share counts the database and the API server too.
78+
`COMPUTER_MEMORY_HIGH_PERCENT=0` and `COMPUTER_TAB_IDLE_MS=0` (Helm: `computers.memoryHighPercent`
79+
and `computers.tabIdleMs`) keep the old behaviour.
80+
81+
### A computer closes browsers nobody is using when its memory runs high
82+
83+
A computer runs one browser for each Bot that is browsing, and the only limits were a count and half
84+
an hour untouched. Neither looked at memory, so a few heavy pages could take a container past its
85+
memory limit well under the count, and the kernel then killed processes without regard to which Bot
86+
was busy. On Kubernetes it kills everything in the container at once, so every Bot's browser closed
87+
together and nothing could reach the computer until it came back. A computer now reads its own
88+
memory the way the kubelet does (the working set, which leaves out the file cache the kernel can
89+
drop), before it starts a browser and once a minute. At or above 85% of its limit it closes browsers
90+
one at a time, least recently used first, until it is under 75%, so the next launch does not cross
91+
the line again straight away. Launches that arrive together share one pass rather than closing a
92+
browser each. A Bot whose browser was closed starts again where it left off on its next step, still
93+
signed in, because its profile is on disk.
94+
95+
Only a browser nobody is using is closed. Never closed for memory: the browser being started, the
96+
most recently used one, any browser used in the last minute (a Bot between two steps asks for
97+
nothing while its model works out the next), a person's own browser, cloud or native, whose Recorder
98+
session would end, and the browser of a Bot whose wheel a person holds or has been asked to take.
99+
When every browser is in use, none is closed.
100+
101+
This is a check, not a guarantee. Memory is read before each launch and once a minute, so a page
102+
that grows between two checks can still run the container out of memory. A container with no memory
103+
limit is measured against the machine's memory, and a computer outside a container, such as OpenBot
104+
Desktop's, closes nothing for memory. In the all-in-one image the database and the API server run in
105+
the same container as the browsers, and the share counts them too. That is deliberate: the kernel's
106+
out-of-memory kill counts everything in the container and can take the database down, and closing a
107+
browser nobody is using first is how the database is protected. Each close is logged as
108+
`computer-browser-closed-for-memory` with the working set and the limit in bytes.
109+
`COMPUTER_MEMORY_HIGH_PERCENT` (Helm: `computers.memoryHighPercent`) sets the share where closing
110+
starts, and `0` switches it off.
111+
112+
The tabs a Bot leaves behind are closed too. A Bot always works in its newest page, so an older tab,
113+
such as the page a sign-in popup was opened from or a link a site opened in a new tab, is one it
114+
cannot go back to, and it held memory for as long as the browser stayed up. A background tab that
115+
nothing has used, navigated or brought to the front for ten minutes is now closed. The page the Bot
116+
is on is never closed, and neither is any tab of a person's own browser or of a Bot whose wheel a
117+
person holds. `COMPUTER_TAB_IDLE_MS` (Helm: `computers.tabIdleMs`) sets the time, and `0` keeps them.
118+
119+
Through Helm, `computers.browserIdleMs` and `computers.maxBrowsers` now reach the computer as
120+
written. An hour (`browserIdleMs: 3600000`) used to render as `3.6e+06`, which the computer ignored
121+
for its default of thirty minutes, and `browserIdleMs: 0` (keep browsers open) was dropped. All four
122+
of these settings now also reach the computers `computers.mode: sandbox` makes, which had none of
123+
them.
75124

76125
### Generated interfaces use A2UI 1.0
77126

@@ -140,6 +189,12 @@ The newest come first, and a press on When turns the order round. A search box a
140189
Started by buttons narrow the events already loaded, and the count says how many of them match. A
141190
row about a Bot (a message routed to it, say) now names it under Bot.
142191

192+
### The desktop setup page does not freeze when it loads during an install
193+
194+
If the setup page loaded again while the desktop app downloaded local software, the window showed a
195+
spinning cursor until the download finished. Its checks for a running OpenBot and for the last
196+
failure now wait in the background, so the window keeps answering.
197+
143198
### Each Bot on a shared computer has its own files
144199

145200
A computer that serves many Bots kept one folder of files for all of them, so a note one Bot saved
@@ -425,6 +480,13 @@ apps, which nobody could connect there. An app's own page, reached from an older
425480
available yet and that an administrator has to finish setting it up, instead of naming the setting.
426481
The administrator's page for the app still names `COMPOSIO_API_KEY`.
427482

483+
### A new desktop install of v0.2.0 gets past Install again
484+
485+
A desktop app built from `main` stopped every new install at "Installing OpenBot" with
486+
`shared/package.json: No such file or directory`. It had started reading `shared`, `toolkit` and
487+
`extension` as packages, which v0.2.0 does not have: there `shared` is plain source and there is no
488+
`toolkit` or `extension`. It now reads them only when the release includes them.
489+
428490
### A built-in Bot behind an OpenAI proxy answers again
429491

430492
A built-in Bot behind a gateway or proxy in front of OpenAI (an `OPENAI_BASE_URL` that is not OpenAI or
@@ -559,6 +621,53 @@ not visit during a recording are never recorded, and nothing on the browser's ow
559621
settings) or the Recorder's own is. A tab that reloaded as you switched to it no longer loses the saves
560622
it sends.
561623

624+
### Record a task in a Bot's own browser, with nothing to install
625+
626+
A Bot's screen, opened larger from a conversation, is now a browser a person can work in: tabs, an
627+
address bar, back, forward and reload sit above the page. Take control and the bar is yours (typing an
628+
address, switching, opening and closing tabs, all from the keyboard too); while the Bot has control it
629+
shows where the Bot is, read only. The OpenBot Recorder is already in every Bot's browser, idle until
630+
someone records, so recording a task needs no extension and no Connect step.
631+
632+
- **Record replaces "Teach a browser workflow".** Name the task, press Record, do it, press Stop. The
633+
panel shows the name, the time and the step count, asks about narration before your first recording
634+
(the same choice as the Browser page), and after Stop says what was saved to your Learning space. The
635+
recording lands in Intelligence exactly as the Recorder sends from any browser, in your own Learning
636+
space, so every Bot you use learns from it, this one included.
637+
- **Only you are recorded, never the Bot.** Record works only while you hold control, and handing it
638+
back (or the Bot's request ending) stops the recording, with its last steps sent before the Recorder
639+
disconnects; the Bot acts again only once the recording has ended. Taking control now records who
640+
took it, so on a shared Bot only that person can use the bar or record, someone else's recording
641+
shows only that one is running, and nobody else sees its name, its step count or its microphone.
642+
- **Record needs a computer the Bot does not share.** Where every Bot runs on one computer (Helm's
643+
shared mode, or `AGENT_COMPUTER_URL` with no supervisor), Record is refused in every Bot's browser
644+
and the panel says why: your Recorder connection would sit in the Bot's profile, where any other
645+
Bot's shell could read it. The bar still works. The computer refuses it too. With a supervisor
646+
(laptop, Desktop) or a Sandbox, each Bot has its own computer and Record works.
647+
- **One connection per recording.** Record mints a Recorder connection for the person who pressed it
648+
and Stop revokes it; Settings, Browsers lists it as "<Bot>'s browser" while it lives, and says
649+
"recording now" for any browser that is recording.
650+
- While a recording runs in a Bot's browser, that Bot's network policy also lets the Recorder reach
651+
OpenBot and the Intelligence gateway, on their own ports only; the Bot's own policy returns when it
652+
ends.
653+
- **Removed: the old "Teach a browser workflow" panel and everything behind it** (its routes under
654+
`/api/demonstrations`, its instant skill draft, "Save and use with this Bot" and "Run on a schedule").
655+
Skills already saved from it stay, as ordinary skills; its recordings stay in the database, unread.
656+
- When you take control yourself, the line under the screen no longer repeats your own request back as
657+
if you had said it.
658+
- **What you type while you hold control stays yours.** After you hand back, the Bot reads a field you
659+
filled as "filled in by the person", never its value, in its page snapshot, its page text and its
660+
screenshots; a field you only tabbed through keeps the Bot's own value.
661+
- **A Bot's request for help shows in the conversation**, with its reason and Take control, and folds
662+
into one line once answered. After you hand back, the Bot is told what you did and carries on, saying
663+
what it does next; if it still stops without a word, the card says so and offers Carry on.
664+
- **Clear browser data.** One button beside reload, in a Bot's browser and in your own, asks "Clear this
665+
browser?" and wipes it: cookies, saved passwords, history, site storage and the screenshots kept of its
666+
pages. It starts fresh, signed out of everything; OpenBot's own Passwords stay. A Bot's browser can be
667+
cleared only by the person holding its control. Audited as `computer.browser_cleared`.
668+
- Leaving a conversation while you hold a Bot's control no longer leaves it blank: the Bot's reply after
669+
you hand back shows when you return, with no "A browser task is paused" notice.
670+
562671
### Your own browser, opened from OpenBot, with the Recorder already in it
563672

564673
Wherever OpenBot can run a browser for you, the sidebar has a **Browser** button below Connections. It
@@ -619,7 +728,7 @@ close it, and the reason is in the server's and the computer's logs.
619728
- A person's browser always runs the Recorder that came with its OpenBot, even in a profile that ran an
620729
earlier one, and keeps it pinned to the toolbar once.
621730
- A Recorder that is slow to answer is refused instead of taking down the computer and every browser
622-
on it. Typing on a live screen that records a demonstration reaches the page in the order typed.
731+
on it. Typing on a live screen reaches the page in the order typed.
623732
- Opening the Browser page twice in quick succession (reloading it while your browser is still
624733
opening, for one) opens your browser once, connected; before, both opens could be refused, most
625734
often the first after the Recorder was updated. When a browser cannot be opened, the server's and
@@ -639,6 +748,16 @@ close it, and the reason is in the server's and the computer's logs.
639748
Clicking or focusing the desktop window while it downloaded local software showed a spinning cursor
640749
until the download finished. The window now comes forward at once and keeps the setup screen.
641750

751+
### A reinstall no longer picks up the previous installation's Bot files and sign-ins
752+
753+
A desktop installation set up again after its folder was deleted reused whatever the previous one
754+
left in the container engine: Bot workspace files, and each Bot's browser profile with its website
755+
sign-ins, under the same Bot ids. Only the database was checked. Start now finds every volume and
756+
stopped container the previous installation left and offers Start fresh, which deletes exactly those
757+
and then starts; nothing is reused without asking. To keep the data, restore the previous folder
758+
instead. A container from it that is still running stops the deletion until it is stopped. An
759+
installation with its saved key keeps its data as before.
760+
642761
### A new desktop install starts its server again
643762

644763
A desktop install of v0.2.0 stopped at start with `server stopped straight away (exit status: 1)` and

‎agent-computer/Dockerfile‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -56,14 +56,14 @@ RUN bun install --frozen-lockfile
5656

5757
COPY agent-computer/src ./src
5858
# index.ts imports ../../shared/file-download at runtime (shared/computer-control is type-only and
59-
# erased), demonstration.ts imports ../../shared/demonstration-rules, and the browser routes import
60-
# ../../shared/openbot-browser, so the files they need go where those paths resolve from /app/src.
59+
# erased), and the browser routes import ../../shared/openbot-browser and ../../shared/bot-browser,
60+
# so the files they need go where those paths resolve from /app/src.
6161
COPY shared/file-download.ts /shared/file-download.ts
62-
COPY shared/demonstration-rules.ts /shared/demonstration-rules.ts
63-
# The OpenBot Browser's contract, which the browser routes check requests against. It imports zod,
64-
# and a module resolves its imports from its own directory upward, which from /shared never reaches
65-
# /app/node_modules: so /shared is given that tree to resolve from, rather than a second install.
66-
COPY shared/openbot-browser.ts /shared/openbot-browser.ts
62+
# The OpenBot Browser's contract and a Bot's browser's, which the browser routes check requests
63+
# against. They import zod, and a module resolves its imports from its own directory upward, which
64+
# from /shared never reaches /app/node_modules: so /shared is given that tree to resolve from, rather
65+
# than a second install.
66+
COPY shared/openbot-browser.ts shared/bot-browser.ts /shared/
6767
COPY shared/recorder-extension.ts shared/recorder-extension.json /shared/
6868
RUN ln -s /app/node_modules /shared/node_modules
6969

0 commit comments

Comments
 (0)