Skip to content

test(researcher): Playwright e2e suite and CI job - #680

Open
nikola0x0 wants to merge 1 commit into
fix/researcher-title-model-chat-crashfrom
feat/researcher-e2e-ci
Open

test(researcher): Playwright e2e suite and CI job#680
nikola0x0 wants to merge 1 commit into
fix/researcher-title-model-chat-crashfrom
feat/researcher-e2e-ci

Conversation

@nikola0x0

Copy link
Copy Markdown
Collaborator

Summary

Closes the "e2e test researcher" half of the task: a 15-test Playwright suite plus a researcher-e2e CI job, mirroring the existing chatbot-e2e setup.

Researcher had no automated tests at all — the app with the only real Walrus writes was verified entirely by hand. That's how the title-model P0 in #673 reached production.

Stacked on #673 (base fix/researcher-title-model-chat-crash). The P0 regression spec asserts the error-frame guard from that PR, so merge #673 first; this PR then retargets to dev automatically.

The test that would have caught the P0

e2e/chat-stream.test.ts asserts the raw SSE frames from /api/chat: a failing title model must not put an error frame on the stream.

Verified as a real regression test, not merely a passing one — with the try/catch guard removed it fails with the exact production symptom, while the answer still streams underneath:

Expected length: 0
Received length: 1
Received array:  [{"errorText": "Oops, an error occurred!", "type": "error"}]

Mock seams

Four seams, all keyed off the existing isTestEnvironment, so a run never reaches OpenRouter, Sui, or the relayer:

Seam What it does
lib/ai/providers.ts Maps picker model ids onto the three registered mock models. The pre-existing test branch was broken — it passed the raw id straight through, so languageModel("google/gemini-2.5-flash") would have thrown NoSuchModelError on the first message. The branch had simply never run.
lib/ai/models.mock.ts A FAIL_TITLE_GENERATION sentinel in a user message makes the title model reject, reproducing the P0 on demand.
lib/auth/delegate-account.mock.ts (new) Fabricates the MemWalAccount object for two fixture identities. The real binding validation still runs, so unregistered keys and unknown accounts fail exactly as they do on-chain — both asserted.
lib/sprint/memwal.ts MemWalMock from the SDK, one instance per process, so remember → recall round-trips in memory and CI can never write to the production relayer.

Coverage

Spec Covers
auth.setup.ts Signs in as account A through the real form (doubles as the happy-path login test) and account B via API; saves both storage states.
e2e/auth.test.ts Anonymous redirects; unregistered key, unknown account, malformed key rejections.
e2e/chat.test.ts Input/suggestions render; message streams a reply; chat survives reload with both rows. Asserts zero uncaught page errors, guarding the auto-resume TypeError from 39e102ae.
e2e/chat-stream.test.ts SSE frame protocol + the P0 regression test above.
e2e/visibility.test.ts Private/Public boundary across two real accounts.

Two behaviors pinned as they are, not as expected

Both found while writing the specs; neither leaks content:

  1. A private chat opened by another user returns HTTP 200, not 404 — app/(chat)/chat/[id]/page.tsx wraps the async component in <Suspense>, so the streaming shell is flushed before notFound() runs. The user correctly sees the not-found page; only the status code is misleading (matters for monitoring/bots). The spec asserts the rendered page and the absence of the canary.
  2. Anonymous visitors are redirected to /login even for public chats, since proxy.ts's auth check precedes the visibility check — the known "Public — Anyone with the link" product gap. Pinned deliberately, with a comment to update it if that product decision changes.

Flakiness worked out up front

global-setup.ts clears the auth limiter's Redis keys before each run. The limiter allows 10 verify attempts per IP per minute and a run spends five, so without this a second run inside a minute fails at sign-in with a 429 that looks nothing like the real problem. Both identities also sign in once in a setup project rather than per-test, keeping CI's 2 retries clear of the limit.

Also worth knowing for anyone adding specs (documented in tests/README.md): browser.newContext() inherits the project's use.storageState, so a context that must be someone else — including an anonymous one — has to pass its own explicitly. Miss it and the test still passes, for the wrong reason. My first draft of the visibility specs hit exactly this.

Verification

  • 15/15 green on three consecutive runs, ~10s each
  • Green from cold with Playwright managing its own webServer, and with AUTH_SECRET / NEXT_PUBLIC_MEMWAL_PACKAGE_ID unset — so the config defaults CI relies on are exercised
  • Negative control: guard removed → P0 spec fails as shown above; restored → green
  • tsc --noEmit clean; pnpm test:unit 14/14

Notes

  • Lint is not wired into the new job: ultracite/biome version mismatch already breaks pnpm lint repo-wide (soft-failed for chatbot in the same workflow). Same follow-up.
  • The suite has no live-model or live-Walrus canary by design; the real remember → recall loop against the production relayer stays a manual check.

Researcher had no automated tests — the app with the only real Walrus
writes was verified entirely by hand. This adds a 15-test Playwright
suite and wires it into CI as `researcher-e2e`, mirroring the existing
`chatbot-e2e` job.

The suite's reason for existing is the first spec: the retired
title-model P0 (c031c47) shipped because nothing would have caught it.
`chat-stream.test.ts` now asserts the raw SSE frames directly — a
failing title model must not put an `error` frame on the stream.
Verified as a real regression test, not just a passing one: with the
try/catch guard removed the spec fails with the exact production
symptom, `{"type":"error","errorText":"Oops, an error occurred!"}`,
while the answer still streams underneath.

Four mock seams, all keyed off the existing `isTestEnvironment`, so a
run never reaches OpenRouter, Sui, or the relayer:

- `providers.ts` — picker model ids now map onto the three registered
  mock models. The pre-existing test branch passed the raw id straight
  through, so `languageModel("google/gemini-2.5-flash")` would have
  thrown NoSuchModelError on the first message; the branch had simply
  never run.
- `models.mock.ts` — a `FAIL_TITLE_GENERATION` sentinel in a user
  message makes the title model reject, reproducing the P0 on demand.
- `delegate-account.mock.ts` — fabricates the `MemWalAccount` object
  for two fixture identities. The real binding validation still runs,
  so unregistered keys and unknown accounts fail exactly as they do
  on-chain (both asserted).
- `memwal.ts` — `MemWalMock` for the Walrus client, one instance per
  process, so CI can never write to the production relayer.

Coverage: delegate-key login (form and rejections), streamed replies,
reload persistence with zero uncaught page errors (guards the
auto-resume TypeError from 39e102a), SSE frame protocol, and the
Private/Public boundary across two real accounts.

Two behaviors pinned as they actually are rather than as expected:
a private chat opened by another user renders Next's not-found page
with HTTP 200, because `<Suspense>` flushes the shell before
`notFound()` runs; and anonymous visitors are redirected to /login even
for public chats, since the auth check precedes the visibility check.
Neither leaks content. The second is the known "Anyone with the link"
product gap.

`global-setup.ts` clears the auth limiter's Redis keys before each run.
The limiter allows 10 verify attempts per IP per minute and a run
spends five, so without it a second run inside a minute fails at
sign-in with a 429 that looks nothing like the real problem. Both
identities also sign in once in a setup project rather than per-test,
which keeps CI retries clear of the limit.

Verified locally: 15/15 green on three consecutive runs (~10s each),
from cold with Playwright managing its own webServer and with
AUTH_SECRET and the package id unset, so the config defaults used by CI
are exercised.
ducnmm pushed a commit that referenced this pull request Aug 20, 2026
Noter had zero automated tests; the on-chain registration flow, delegate-key
auth, and note CRUD were verified entirely by hand. 22 specs across app
shell, auth, note lifecycle, and the memory API contract, running against a
real Next.js dev server and a fresh Postgres.

Mock seam for the new delegate-account binding check
------------------------------------------------------
connectDelegateKey now calls assertDelegateAccountBinding (a separate,
already-merged change), which reads the claimed account off-chain via gRPC
and rejects any key that isn't registered in its delegate_keys list — so a
random, never-registered key/account pair can no longer reach an
authenticated session the way it could before that change landed. Mirroring
researcher's PR #680 pattern, delegate-account.ts now branches on
lib/constants.ts's isTestEnvironment (set by playwright.config.ts passing
PLAYWRIGHT=True to the webServer) and serves a fixture object from
delegate-account.mock.ts instead of the gRPC read, so the real validation
logic still runs meaningfully: an unknown account or an unregistered key
fails the exact same way it would on-chain.

Noter authenticates a fresh identity per test rather than reusing two shared
identities across a whole run (researcher's approach) — with `workers: 2` and
~15 login call sites, two fixed identities would have concurrent tests
collide on each other's notes. delegate-account.mock.ts and
fixtures/delegate-key.ts instead generate the same 24-entry deterministic
pool independently (index N -> accountId byte N repeated, privateKey byte
N+0x40 repeated), and the test fixture hands out a never-yet-used entry per
call, interleaved by Playwright's parallelIndex so two worker processes
never claim the same one. public_key is stored base64, not hex: the binding
check's parser tries fromBase64() before falling back to raw hex, and every
64-char hex string (alphabet 0-9a-f, always length-divisible-by-4) also
happens to be valid-but-wrong base64, so a hex value there silently decodes
to the wrong bytes instead of ever matching.

The memory-write specs assert against the real relayer response for a
fixture (unregistered) key, so there's no live-Walrus canary in this suite by
design, same as #680 documents for researcher: the real remember -> recall
round trip against production Walrus Memory stays a manual check.

CI job
------
noter-e2e mirrors chatbot-e2e's shape (Postgres service container, cached
Playwright browsers, report/trace upload on failure). noter-checks adds
tsc --noEmit and a full `next build` so a type or build regression fails CI
even on a change the e2e specs don't happen to cover.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant