test(sdk): e2e suite and CI workflow against the dev relayer - #679
test(sdk): e2e suite and CI workflow against the dev relayer#679nikola0x0 wants to merge 4 commits into
Conversation
Mirrors the Python SDK's tests/test_integration.py: no-auth health, compatibility and auth-rejection checks that always run, plus authenticated remember/recall/analyze/restore coverage (and the JS-only bulk, manual-mode and embed surfaces) that skips without MEMWAL_PRIVATE_KEY / MEMWAL_ACCOUNT_ID. Authenticated writes land in a per-run sdk-e2e-<random> namespace so the shared bench account's real namespaces stay clean, and remember waits get 120s of headroom over the SDK's 60s default (a live write measures ~44s on dev). Named *.e2e.mjs so the offline unit glob (test/**/*.test.mjs) never picks the suite up; run it via the new test:e2e script. [WALM-353]
Sibling of test-python-sdk.yml. The unit job runs the offline suite on Node 22 and 24 for every PR touching packages/sdk — closing the gap where SDK unit tests never ran in CI at all. The e2e job reuses the benchmark-dev environment credentials, is limited to dev pushes, manual dispatch and a weekly cron (PRs are excluded: fork PRs get no environment secrets, and every authenticated run writes real memories), fails loudly when credentials are missing instead of green-skipping, and uploads a junit artifact plus a run summary. The cron sits 30 minutes after the Python suite's slot so the two weekly runs don't write through the shared bench account at the same time. [WALM-353]
Review of the authenticated half — which cannot run locally without the bench credentials — caught two tests that were guaranteed to go red on the first CI run: - embed() POSTs /api/embed, which is absent from the relayer's protected route table entirely. - rememberManual() sends blob_id, but RememberManualRequest requires encrypted_data and has no blob_id field, so axum 422s the call before the handler sees it. Both methods are covered only by MemWalMock today, which is why the drift went unnoticed. The suite documents the omission inline rather than deleting it silently; the SDK bug is tracked in WALM-371. Also hardens the remaining live tests: the analyze fan-out gets twice the single-write budget (N facts, one wallet-job pipeline each, on a contended shared account), the one-shot status probe tolerates a background 'failed' since it asserts acceptance rather than pipeline health, and MEMWAL_REMEMBER_TIMEOUT_MS now rejects unparseable input instead of silently becoming NaN. [WALM-353]
|
Pushed review fixes (e2fb815). A pre-review pass over the authenticated half — which can't run locally without the bench credentials — found two tests that would have gone red on the first CI run, both because the SDK method itself targets a contract the relayer doesn't serve:
Both are covered only by Also hardened the remaining live tests: the Verification after the changes: unit 66 pass / 0 fail; no-auth e2e slice against the dev relayer 7 pass / 11 skipped / 0 fail. |
The dev-only restriction was inherited from test-python-sdk.yml, whose comment justified it as 'staging and production follow once their credentials exist'. That premise no longer holds: benchmark-staging and benchmark-production both carry BENCH_DELEGATE_KEY, BENCH_ACCOUNT_ID and BENCH_SERVER_URL today. Each long-lived branch now tests the deployment it corresponds to — dev, staging, main -> production — and workflow_dispatch takes an explicit environment choice, following the selection pattern benchmark-live.yml already uses. Scheduled runs stay on dev, since cron fires on the default branch, so production is only reached by an explicit main push or a deliberate dispatch. Drops the hardcoded relayer URL fallback. With one environment a literal default guarded against an unset variable; with three it becomes the hazard it was meant to prevent, since a production run with a missing variable would silently exercise dev. The credential check now requires MEMWAL_SERVER_URL and fails the job without it. The suite is safe to point at production because every authenticated write is confined to a per-run sdk-e2e-<random> namespace; the only reference to 'default' is the body of the 401-rejection tests, which never write. [WALM-353]
Closes WALM-353. Sibling of #661 (Python SDK e2e) — same structure, env contract and credentials, so the two suites stay comparable.
What
packages/sdk/test/e2e/live.e2e.mjs(+test:e2escript) — a live-relayer suite in two halves:/health, the compatibility contract (GET /versionaccepts this SDK), unsigned → 401, wrong signature → 401, expired/future timestamp → 401, unregistered key surfaced as an auth error by the SDK.MEMWAL_PRIVATE_KEY/MEMWAL_ACCOUNT_ID: remember acceptance +getRememberStatus,rememberAndWait, namespace fallback/override,recall,analyze/analyzeAndWait,rememberBulkAndWait,restore, and a full remember → recall → verify loop.Hygiene mirrors the Python suite: every authenticated write lands in a per-run
sdk-e2e-<random>namespace, and remember waits get 120s of headroom (a live write measures ~44s on dev), with theanalyzefan-out given twice that since each extracted fact is its own write pipeline.ask()has no JS-SDK equivalent — its analogue is theai/integration, which needs a model provider and is out of scope here.Deliberately not covered —
embed()and lightweight manual mode. Both SDK methods target contracts this relayer does not serve:/api/embedis absent from the protected route table, andrememberManual()sendsblob_idwhereRememberManualRequestrequiresencrypted_data(422 before the handler). Tests for them would be guaranteed red. Documented inline and tracked in WALM-371 — a real find this effort surfaced, since both methods are covered only byMemWalMocktoday..github/workflows/test-sdk.yml— sibling oftest-python-sdk.yml:unit: the offline suite on Node 22 + 24 for every PR touchingpackages/sdk/**. This also closes a standing gap: the SDK's unit tests were not run by any CI workflow before this.e2e: each long-lived branch tests the deployment it corresponds to —dev→benchmark-dev,staging→benchmark-staging,main→benchmark-production— plusworkflow_dispatchwith an explicit environment choice, following the selection patternbenchmark-live.ymlalready uses. Scheduled runs stay on dev (cron fires on the default branch), so production is only reached by an explicitmainpush or a deliberate dispatch. PRs are excluded: fork PRs get no environment secrets, and every authenticated run writes real memories. No new secrets — all three environments already carryBENCH_DELEGATE_KEY/BENCH_ACCOUNT_ID/BENCH_SERVER_URL.Two notes on the multi-environment setup:
MEMWAL_SERVER_URLand fails the job without it.sdk-e2e-<random>namespace; the only reference todefaultis the body of the 401-rejection tests, which never write. Don't relax that isolation, and don't point this job at an account holding user data.One deliberate deviation from the Python workflow's matrix philosophy: Node 22/24 rather than reaching further back, because
node --testonly accepts glob arguments from Node 21.Test plan
.e2e.mjsfiles)https://relayer.dev.memwal.ai: 7 pass, 11 skipped (credentials absent locally), 0 failworkflow_dispatchafter merge, as with ci(python-sdk): build, unit test and e2e workflow against the dev relayer #661. Worth one run by someone holding thebenchmark-devcredentials before merge if we want it proven earlier —MEMWAL_PRIVATE_KEY=… MEMWAL_ACCOUNT_ID=… MEMWAL_SERVER_URL=https://relayer.dev.memwal.ai pnpm --filter @mysten-incubation/memwal test:e2e