Skip to content

test(sdk): e2e suite and CI workflow against the dev relayer - #679

Open
nikola0x0 wants to merge 4 commits into
devfrom
nikolale/walm-353-e2e-test-js-sdk
Open

test(sdk): e2e suite and CI workflow against the dev relayer#679
nikola0x0 wants to merge 4 commits into
devfrom
nikolale/walm-353-e2e-test-js-sdk

Conversation

@nikola0x0

@nikola0x0 nikola0x0 commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Closes WALM-353. Sibling of #661 (Python SDK e2e) — same structure, env contract and credentials, so the two suites stay comparable.

What

packages/sdk/test/e2e/live.e2e.mjs (+ test:e2e script) — a live-relayer suite in two halves:

  • No-auth, always runs: /health, the compatibility contract (GET /version accepts this SDK), unsigned → 401, wrong signature → 401, expired/future timestamp → 401, unregistered key surfaced as an auth error by the SDK.
  • Authenticated, skips without MEMWAL_PRIVATE_KEY / MEMWAL_ACCOUNT_ID: remember acceptance + getRememberStatus, rememberAndWait, namespace fallback/override, recall, analyze / analyzeAndWait, rememberBulkAndWait, restore, and a full remember → recall → verify loop.

Hygiene mirrors the Python suite: every authenticated write lands in a per-run sdk-e2e-<random> namespace, and remember waits get 120s of headroom (a live write measures ~44s on dev), with the analyze fan-out given twice that since each extracted fact is its own write pipeline.

ask() has no JS-SDK equivalent — its analogue is the ai/ integration, which needs a model provider and is out of scope here.

Deliberately not covered — embed() and lightweight manual mode. Both SDK methods target contracts this relayer does not serve: /api/embed is absent from the protected route table, and rememberManual() sends blob_id where RememberManualRequest requires encrypted_data (422 before the handler). Tests for them would be guaranteed red. Documented inline and tracked in WALM-371 — a real find this effort surfaced, since both methods are covered only by MemWalMock today.

.github/workflows/test-sdk.yml — sibling of test-python-sdk.yml:

  • unit: the offline suite on Node 22 + 24 for every PR touching packages/sdk/**. This also closes a standing gap: the SDK's unit tests were not run by any CI workflow before this.
  • e2e: each long-lived branch tests the deployment it corresponds todevbenchmark-dev, stagingbenchmark-staging, mainbenchmark-production — plus workflow_dispatch with an explicit environment choice, following the selection pattern benchmark-live.yml already uses. Scheduled runs stay on dev (cron fires on the default branch), so production is only reached by an explicit main push or a deliberate dispatch. PRs are excluded: fork PRs get no environment secrets, and every authenticated run writes real memories. No new secrets — all three environments already carry BENCH_DELEGATE_KEY / BENCH_ACCOUNT_ID / BENCH_SERVER_URL.

Two notes on the multi-environment setup:

  • No relayer-URL fallback. With one environment a literal default guarded against an unset variable; with three it becomes the hazard it was meant to prevent, since a production run with a missing variable would silently exercise dev. The credential check now requires MEMWAL_SERVER_URL and fails the job without it.
  • Production safety rests on namespace isolation. Every authenticated write is confined to the per-run sdk-e2e-<random> namespace; the only reference to default is the body of the 401-rejection tests, which never write. Don't relax that isolation, and don't point this job at an account holding user data.

One deliberate deviation from the Python workflow's matrix philosophy: Node 22/24 rather than reaching further back, because node --test only accepts glob arguments from Node 21.

Test plan

  • Unit suite: 66 pass, 0 fail (glob untouched by the new .e2e.mjs files)
  • No-auth e2e slice against https://relayer.dev.memwal.ai: 7 pass, 11 skipped (credentials absent locally), 0 fail
  • Environment-selection expression simulated across all 8 trigger cases (push dev/staging/main, schedule, dispatch × 4)
  • Authenticated e2e: workflow_dispatch after merge, as with ci(python-sdk): build, unit test and e2e workflow against the dev relayer #661. Worth one run by someone holding the benchmark-dev credentials before merge if we want it proven earlier — MEMWAL_PRIVATE_KEY=… MEMWAL_ACCOUNT_ID=… MEMWAL_SERVER_URL=https://relayer.dev.memwal.ai pnpm --filter @mysten-incubation/memwal test:e2e

Mirrors the Python SDK's tests/test_integration.py: no-auth health,
compatibility and auth-rejection checks that always run, plus
authenticated remember/recall/analyze/restore coverage (and the
JS-only bulk, manual-mode and embed surfaces) that skips without
MEMWAL_PRIVATE_KEY / MEMWAL_ACCOUNT_ID. Authenticated writes land in
a per-run sdk-e2e-<random> namespace so the shared bench account's
real namespaces stay clean, and remember waits get 120s of headroom
over the SDK's 60s default (a live write measures ~44s on dev).

Named *.e2e.mjs so the offline unit glob (test/**/*.test.mjs) never
picks the suite up; run it via the new test:e2e script. [WALM-353]
Sibling of test-python-sdk.yml. The unit job runs the offline suite on
Node 22 and 24 for every PR touching packages/sdk — closing the gap
where SDK unit tests never ran in CI at all. The e2e job reuses the
benchmark-dev environment credentials, is limited to dev pushes,
manual dispatch and a weekly cron (PRs are excluded: fork PRs get no
environment secrets, and every authenticated run writes real
memories), fails loudly when credentials are missing instead of
green-skipping, and uploads a junit artifact plus a run summary.

The cron sits 30 minutes after the Python suite's slot so the two
weekly runs don't write through the shared bench account at the same
time. [WALM-353]
Review of the authenticated half — which cannot run locally without
the bench credentials — caught two tests that were guaranteed to go
red on the first CI run:

- embed() POSTs /api/embed, which is absent from the relayer's
  protected route table entirely.
- rememberManual() sends blob_id, but RememberManualRequest requires
  encrypted_data and has no blob_id field, so axum 422s the call
  before the handler sees it.

Both methods are covered only by MemWalMock today, which is why the
drift went unnoticed. The suite documents the omission inline rather
than deleting it silently; the SDK bug is tracked in WALM-371.

Also hardens the remaining live tests: the analyze fan-out gets twice
the single-write budget (N facts, one wallet-job pipeline each, on a
contended shared account), the one-shot status probe tolerates a
background 'failed' since it asserts acceptance rather than pipeline
health, and MEMWAL_REMEMBER_TIMEOUT_MS now rejects unparseable input
instead of silently becoming NaN. [WALM-353]
@nikola0x0

Copy link
Copy Markdown
Collaborator Author

Pushed review fixes (e2fb815). A pre-review pass over the authenticated half — which can't run locally without the bench credentials — found two tests that would have gone red on the first CI run, both because the SDK method itself targets a contract the relayer doesn't serve:

  • embed() POSTs /api/embed, which isn't in the protected route table in services/server/src/main.rs at all (and isn't in docs/relayer/api-reference.md).
  • rememberManual() sends {blob_id, vector, namespace}, but RememberManualRequest (services/server/src/types.rs:1374-1379) requires encrypted_data and has no blob_id field, so axum 422s it before the handler runs. The handler uploads ciphertext to Walrus — the opposite of the lightweight mapping the SDK docstring promises.

Both are covered only by MemWalMock today, which is why the drift went unnoticed. The suite now documents the omission inline instead of dropping it silently, and the SDK bug is tracked in WALM-371. recallManual() is unaffected — its wire contract matches.

Also hardened the remaining live tests: the analyze fan-out gets twice the single-write budget (each fact is its own wallet-job pipeline on a shared account), the one-shot status probe tolerates a background failed since it asserts acceptance rather than pipeline health, and MEMWAL_REMEMBER_TIMEOUT_MS now rejects unparseable input instead of silently becoming NaN.

Verification after the changes: unit 66 pass / 0 fail; no-auth e2e slice against the dev relayer 7 pass / 11 skipped / 0 fail.

The dev-only restriction was inherited from test-python-sdk.yml, whose
comment justified it as 'staging and production follow once their
credentials exist'. That premise no longer holds: benchmark-staging
and benchmark-production both carry BENCH_DELEGATE_KEY,
BENCH_ACCOUNT_ID and BENCH_SERVER_URL today.

Each long-lived branch now tests the deployment it corresponds to —
dev, staging, main -> production — and workflow_dispatch takes an
explicit environment choice, following the selection pattern
benchmark-live.yml already uses. Scheduled runs stay on dev, since
cron fires on the default branch, so production is only reached by an
explicit main push or a deliberate dispatch.

Drops the hardcoded relayer URL fallback. With one environment a
literal default guarded against an unset variable; with three it
becomes the hazard it was meant to prevent, since a production run
with a missing variable would silently exercise dev. The credential
check now requires MEMWAL_SERVER_URL and fails the job without it.

The suite is safe to point at production because every authenticated
write is confined to a per-run sdk-e2e-<random> namespace; the only
reference to 'default' is the body of the 401-rejection tests, which
never write. [WALM-353]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant