Skip to content

fix: Fix purgeOnStart on aliased storages - #4013

Merged
janbuchar merged 4 commits into
v4from
fix-aliased-storage-purging
Aug 12, 2026
Merged

fix: Fix purgeOnStart on aliased storages#4013
janbuchar merged 4 commits into
v4from
fix-aliased-storage-purging

Conversation

@janbuchar

Copy link
Copy Markdown
Contributor

Unnamed storages (default and aliased) are now purged on start, named ones are not — crawlee-python's rule. The fs backend sweeps the storage directories, so leftovers from a previous process are caught too.

Two adjacent bugs, one commit each:

  • createDatasetBackend() / ({}) didn't open the default storage, though StorageIdentifier says they do
  • the __default__ sentinel leaked into the directory name, so default storages lived in storage/datasets/__default__ rather than default

ThrottlingRequestManager (#3741) sub-queues are alias-keyed, so they now only survive a restart with purgeOnStart off — as its docs already said.

`StorageCache` documents `byAlias` as run-scoped, but neither backend purged
an aliased storage, so alias-keyed storages outlived the run like named ones.
Most visibly, `openOwnedRequestQueue` gives every crawler instance after the
first an `__default_<id>__` queue, so a second crawler silently resumed the
previous run's requests even with `purgeOnStart` enabled.

Unnamed storages are now purged and named ones are not, matching
crawlee-python's `_purge_if_needed`. The file-system backend enumerates the
storage directories and reads each `__metadata__.json`, so leftovers from a
previous process are found too; directories with no metadata were not written
by Crawlee and are left alone.
`StorageIdentifier` documents `{}` / omitted as the default storage, but the
backends only applied that rule to the `id` / `name` / `alias` triple they were
handed. `createDatasetBackend()`, `createDatasetBackend({})` and
`createDatasetBackend({ alias: '__default__' })` each returned a different
storage; on the in-memory backend every no-argument call built a fresh uncached
one, so the store could not be reopened at all.

Both backends now resolve "no identifier" to the reserved alias, the same rule
the storage frontends apply in `resolveStorageIdentifier`.
…ectory name

The reserved `__default__` alias was normalized to `default` for the cache key
but handed to the native client verbatim, which names the directory after it. So
the default storages lived in `storage/datasets/__default__` and friends, where
the docs, the project templates and the e2e tooling all say `default`. Every
`storage/` directory written by an earlier version was orphaned, and the
documented way of feeding a local run — dropping an `INPUT.json` into
`storage/key_value_stores/default` — silently read back nothing.

Normalizing in `resolveStorageKey` keeps the cache key and the directory name
in step, so the sentinel now stops at the backend boundary.
@janbuchar janbuchar added the t-tooling Issues with this label are in the ownership of the tooling team. label Aug 11, 2026
@janbuchar
janbuchar requested a review from barjin August 11, 2026 17:35

@barjin barjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we give this a more prominent place in the docs somewhere? Do I understand right that

const cheerioQueue = await RequestQueue.open({ alias: 'cheerio' });

const browserCrawler = new PlaywrightCrawler(
    // queue: default
    (enqueueLinks) => {
        enqueueLinks({queue: cheerioQueue})
    }
);
await browserCrawler.run([...]);

const cheerioCrawler = new CheerioCrawler(cheerioQueue);
await cheerioCrawler.run();

This will fail as the cheerioQueue gets purged on cheerioCrawler.run()? If so, we should imo have a separate docs example / guide on how to run multiple crawlers, as this is a fairly popular pattern afaik.

nvm, I see that calling purgeDefaultStorages from crawler.run() will call backend.purge() only once 👍

I suppose this is also different from the on-platform behaviour, right?

@janbuchar

Copy link
Copy Markdown
Contributor Author

I suppose this is also different from the on-platform behaviour, right?

Purging isn't really a thing on Apify platform, you get fresh default storages at the start of each run and aliased storages need to be either created, or mapped to IDs of storages created according to the Actor output config.

@janbuchar
janbuchar merged commit 17208a7 into v4 Aug 12, 2026
8 checks passed
@janbuchar
janbuchar deleted the fix-aliased-storage-purging branch August 12, 2026 12:05
B4nan added a commit that referenced this pull request Aug 12, 2026
…age layout

- drop the migration-narration comments the review flagged (and their copies in
  sibling actors), keep only comments explaining non-obvious behavior
- link the LOCAL matrix TODO to #3406 and the type-unsafe enqueueLinks casts
  to #4024
- un-rename the gotoOptions destructuring in the store-pagination actors
- load the fs-storage native binding via a lazy dynamic import instead of
  createRequire, as suggested in review
- read e2e storages from the 'default' directories: since #4013 the default
  alias maps to the 'default' directory on disk again
- move the logger and Cloudflare sections of the upgrading guide into the
  conditional sections
B4nan added a commit that referenced this pull request Aug 12, 2026
Gets the E2E test suite running against v4. The suite hadn't been run
since the v4 rewrite and everything failed on startup. After these
changes the MEMORY run passes locally end to end, and getting there
surfaced a few real regressions in the packages themselves.

## Package fixes

- `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global
`document` (which doesn't exist in Node) instead of the parsed window,
so every call crashed at runtime.
- `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a
v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed
and `firstErrorHtmlUrl` never made it into the crawler statistics.
- `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override
`enqueueLinks`, exposing the strict urls-required signature even though
the runtime helper extracts URLs from the parsed document.
- `LinkeDOMCrawler` can now be constructed without arguments, like the
other crawlers.

## E2E suite changes

- Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports
`snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists
there).
- Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless
key-value records; the short-lived `__default__` directory alias it
originally targeted was a bug, fixed in #4013) and to the
`@crawlee/utils` exports split.
- Migrated test actors to the v4 APIs: the `logger` option with
`ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the
crawling context, `session.setCookie()`, a custom `SessionPool` instead
of `sessionPoolOptions`, the WHATWG `Response` returned by
`sendRequest`, `registerDeferredCleanup` for dataset writes that must
survive a throwing handler, and explicit enqueue strategies now that
`include` globs are ANDed with the default same-hostname strategy.
- The ignore-ssl test now configures TLS verification on the http
client, because the crawler-level `ignoreSslErrors` option is not wired
to the default client in v4. That dangling option deserves a separate
fix or removal, since it currently does nothing.
- The impit test pins session fingerprints, since the random default
fingerprint overrides the client's browser impersonation.
- Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the
ES lib and broke compilation on `ErrorOptions`).
- Skipped the zero-concurrency queue test: it stages a stuck queue
through the v3 client-side `inProgress` set, which the rewritten queue
doesn't have.
- Fixed the camoufox fetch retry loop fetching 5x even on success, and
removed a duplicate `apify` dependency key that silently downgraded the
curl-impersonate actor to SDK v3.
- Commented out the LOCAL storage matrix entry in the workflow, as
`@apify/storage-local` doesn't support v4.
B4nan pushed a commit that referenced this pull request Aug 12, 2026
- closes #3998

Unnamed storages (default and aliased) are now purged on start, named
ones are not — crawlee-python's rule. The fs backend sweeps the storage
directories, so leftovers from a previous process are caught too.

Two adjacent bugs, one commit each:
- `createDatasetBackend()` / `({})` didn't open the default storage,
though `StorageIdentifier` says they do
- the `__default__` sentinel leaked into the directory name, so default
storages lived in `storage/datasets/__default__` rather than `default`

`ThrottlingRequestManager` (#3741) sub-queues are alias-keyed, so they
now only survive a restart with `purgeOnStart` off — as its docs already
said.
B4nan added a commit that referenced this pull request Aug 12, 2026
Gets the E2E test suite running against v4. The suite hadn't been run
since the v4 rewrite and everything failed on startup. After these
changes the MEMORY run passes locally end to end, and getting there
surfaced a few real regressions in the packages themselves.

- `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global
`document` (which doesn't exist in Node) instead of the parsed window,
so every call crashed at runtime.
- `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a
v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed
and `firstErrorHtmlUrl` never made it into the crawler statistics.
- `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override
`enqueueLinks`, exposing the strict urls-required signature even though
the runtime helper extracts URLs from the parsed document.
- `LinkeDOMCrawler` can now be constructed without arguments, like the
other crawlers.

- Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports
`snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists
there).
- Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless
key-value records; the short-lived `__default__` directory alias it
originally targeted was a bug, fixed in #4013) and to the
`@crawlee/utils` exports split.
- Migrated test actors to the v4 APIs: the `logger` option with
`ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the
crawling context, `session.setCookie()`, a custom `SessionPool` instead
of `sessionPoolOptions`, the WHATWG `Response` returned by
`sendRequest`, `registerDeferredCleanup` for dataset writes that must
survive a throwing handler, and explicit enqueue strategies now that
`include` globs are ANDed with the default same-hostname strategy.
- The ignore-ssl test now configures TLS verification on the http
client, because the crawler-level `ignoreSslErrors` option is not wired
to the default client in v4. That dangling option deserves a separate
fix or removal, since it currently does nothing.
- The impit test pins session fingerprints, since the random default
fingerprint overrides the client's browser impersonation.
- Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the
ES lib and broke compilation on `ErrorOptions`).
- Skipped the zero-concurrency queue test: it stages a stuck queue
through the v3 client-side `inProgress` set, which the rewritten queue
doesn't have.
- Fixed the camoufox fetch retry loop fetching 5x even on success, and
removed a duplicate `apify` dependency key that silently downgraded the
curl-impersonate actor to SDK v3.
- Commented out the LOCAL storage matrix entry in the workflow, as
`@apify/storage-local` doesn't support v4.
B4nan pushed a commit that referenced this pull request Aug 18, 2026
- closes #3998

Unnamed storages (default and aliased) are now purged on start, named
ones are not — crawlee-python's rule. The fs backend sweeps the storage
directories, so leftovers from a previous process are caught too.

Two adjacent bugs, one commit each:
- `createDatasetBackend()` / `({})` didn't open the default storage,
though `StorageIdentifier` says they do
- the `__default__` sentinel leaked into the directory name, so default
storages lived in `storage/datasets/__default__` rather than `default`

`ThrottlingRequestManager` (#3741) sub-queues are alias-keyed, so they
now only survive a restart with `purgeOnStart` off — as its docs already
said.
B4nan added a commit that referenced this pull request Aug 18, 2026
Gets the E2E test suite running against v4. The suite hadn't been run
since the v4 rewrite and everything failed on startup. After these
changes the MEMORY run passes locally end to end, and getting there
surfaced a few real regressions in the packages themselves.

- `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global
`document` (which doesn't exist in Node) instead of the parsed window,
so every call crashed at runtime.
- `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a
v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed
and `firstErrorHtmlUrl` never made it into the crawler statistics.
- `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override
`enqueueLinks`, exposing the strict urls-required signature even though
the runtime helper extracts URLs from the parsed document.
- `LinkeDOMCrawler` can now be constructed without arguments, like the
other crawlers.

- Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports
`snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists
there).
- Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless
key-value records; the short-lived `__default__` directory alias it
originally targeted was a bug, fixed in #4013) and to the
`@crawlee/utils` exports split.
- Migrated test actors to the v4 APIs: the `logger` option with
`ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the
crawling context, `session.setCookie()`, a custom `SessionPool` instead
of `sessionPoolOptions`, the WHATWG `Response` returned by
`sendRequest`, `registerDeferredCleanup` for dataset writes that must
survive a throwing handler, and explicit enqueue strategies now that
`include` globs are ANDed with the default same-hostname strategy.
- The ignore-ssl test now configures TLS verification on the http
client, because the crawler-level `ignoreSslErrors` option is not wired
to the default client in v4. That dangling option deserves a separate
fix or removal, since it currently does nothing.
- The impit test pins session fingerprints, since the random default
fingerprint overrides the client's browser impersonation.
- Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the
ES lib and broke compilation on `ErrorOptions`).
- Skipped the zero-concurrency queue test: it stages a stuck queue
through the v3 client-side `inProgress` set, which the rewritten queue
doesn't have.
- Fixed the camoufox fetch retry loop fetching 5x even on success, and
removed a duplicate `apify` dependency key that silently downgraded the
curl-impersonate actor to SDK v3.
- Commented out the LOCAL storage matrix entry in the workflow, as
`@apify/storage-local` doesn't support v4.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants