Conversation
…nd Deno
Most JavaScript crawling is a fetch under another name, and the rest is
axios, got (Crawlee underneath), Puppeteer or Playwright. This gives each of
them a one-line way to pay: present the filed pass on the way out, pay a 402
once, replay with the pass.
wrapFetch(fetch, { key }) any fetch-shaped function
attachX402(axios, client) request + response interceptors
got.extend(x402Hooks(client)) beforeRequest + afterResponse hooks
attachX402(page, client) Puppeteer: page.goto pays at navigation
await attachX402(context, client) Playwright: routed, the page sees the paid answer
None of the adapters import the library they adapt; the package stays two
dependencies. axios and got are tested against the real libraries; the
browser adapters against library-shaped fakes, and by hand with real
Puppeteer on Chrome and on Lightpanda and real Playwright on Chrome.
Two things the real browsers taught. A browser asks for text/html, and a
gateway answers that 402 with its sales page rather than the offer, so every
adapter goes back for the offer as JSON when the 402 carries none. And that
retry must REPLACE the caller's Accept, not add to it: axios sends
`Accept`, the override was `accept`, the Fetch API joined them, and the
gateway served the page again. `acceptingJson` is the fix.
Runtimes: base64 through atob/btoa instead of Buffer, env through
globalThis.process, so the same files run on Node 20.19+, Bun and Deno 2.
CI now runs the suite and the CLI on all three. README rewritten around
the tools people crawl with, with badges. 54 tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSGS7Sy3QmEgTNZDg4eq7g
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
1 participant
What
Adapters so the tools people already crawl with pay as they go:
wrapFetch(fetch, { key })attachX402(axios, client)got.extend(x402Hooks(client))attachX402(page, client)await attachX402(context, client)None import the library they adapt; the package stays at two dependencies.
Runtimes
Bufferand bareprocess.envremoved; CI runs the suite and the CLI on Node 22/24, Bun and Deno 2. README has the badges and a per-tool section.Verified
Two things the real browsers caught and the tests now cover: a browser's 402 is the HTML sales page, not the offer, so adapters go back for it as JSON; and that retry must replace the caller's
Accept, not add a second one.🤖 Generated with Claude Code
https://claude.ai/code/session_01NSGS7Sy3QmEgTNZDg4eq7g