fix: Fix purgeOnStart on aliased storages - #4013
Conversation
`StorageCache` documents `byAlias` as run-scoped, but neither backend purged an aliased storage, so alias-keyed storages outlived the run like named ones. Most visibly, `openOwnedRequestQueue` gives every crawler instance after the first an `__default_<id>__` queue, so a second crawler silently resumed the previous run's requests even with `purgeOnStart` enabled. Unnamed storages are now purged and named ones are not, matching crawlee-python's `_purge_if_needed`. The file-system backend enumerates the storage directories and reads each `__metadata__.json`, so leftovers from a previous process are found too; directories with no metadata were not written by Crawlee and are left alone.
`StorageIdentifier` documents `{}` / omitted as the default storage, but the
backends only applied that rule to the `id` / `name` / `alias` triple they were
handed. `createDatasetBackend()`, `createDatasetBackend({})` and
`createDatasetBackend({ alias: '__default__' })` each returned a different
storage; on the in-memory backend every no-argument call built a fresh uncached
one, so the store could not be reopened at all.
Both backends now resolve "no identifier" to the reserved alias, the same rule
the storage frontends apply in `resolveStorageIdentifier`.
…ectory name The reserved `__default__` alias was normalized to `default` for the cache key but handed to the native client verbatim, which names the directory after it. So the default storages lived in `storage/datasets/__default__` and friends, where the docs, the project templates and the e2e tooling all say `default`. Every `storage/` directory written by an earlier version was orphaned, and the documented way of feeding a local run — dropping an `INPUT.json` into `storage/key_value_stores/default` — silently read back nothing. Normalizing in `resolveStorageKey` keeps the cache key and the directory name in step, so the sentinel now stops at the backend boundary.
There was a problem hiding this comment.
Can we give this a more prominent place in the docs somewhere? Do I understand right that
const cheerioQueue = await RequestQueue.open({ alias: 'cheerio' });
const browserCrawler = new PlaywrightCrawler(
// queue: default
(enqueueLinks) => {
enqueueLinks({queue: cheerioQueue})
}
);
await browserCrawler.run([...]);
const cheerioCrawler = new CheerioCrawler(cheerioQueue);
await cheerioCrawler.run();This will fail as the cheerioQueue gets purged on cheerioCrawler.run()? If so, we should imo have a separate docs example / guide on how to run multiple crawlers, as this is a fairly popular pattern afaik.
nvm, I see that calling purgeDefaultStorages from crawler.run() will call backend.purge() only once 👍
I suppose this is also different from the on-platform behaviour, right?
Purging isn't really a thing on Apify platform, you get fresh default storages at the start of each run and aliased storages need to be either created, or mapped to IDs of storages created according to the Actor output config. |
…age layout - drop the migration-narration comments the review flagged (and their copies in sibling actors), keep only comments explaining non-obvious behavior - link the LOCAL matrix TODO to #3406 and the type-unsafe enqueueLinks casts to #4024 - un-rename the gotoOptions destructuring in the store-pagination actors - load the fs-storage native binding via a lazy dynamic import instead of createRequire, as suggested in review - read e2e storages from the 'default' directories: since #4013 the default alias maps to the 'default' directory on disk again - move the logger and Cloudflare sections of the upgrading guide into the conditional sections
Gets the E2E test suite running against v4. The suite hadn't been run since the v4 rewrite and everything failed on startup. After these changes the MEMORY run passes locally end to end, and getting there surfaced a few real regressions in the packages themselves. ## Package fixes - `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global `document` (which doesn't exist in Node) instead of the parsed window, so every call crashed at runtime. - `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed and `firstErrorHtmlUrl` never made it into the crawler statistics. - `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override `enqueueLinks`, exposing the strict urls-required signature even though the runtime helper extracts URLs from the parsed document. - `LinkeDOMCrawler` can now be constructed without arguments, like the other crawlers. ## E2E suite changes - Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports `snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists there). - Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless key-value records; the short-lived `__default__` directory alias it originally targeted was a bug, fixed in #4013) and to the `@crawlee/utils` exports split. - Migrated test actors to the v4 APIs: the `logger` option with `ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the crawling context, `session.setCookie()`, a custom `SessionPool` instead of `sessionPoolOptions`, the WHATWG `Response` returned by `sendRequest`, `registerDeferredCleanup` for dataset writes that must survive a throwing handler, and explicit enqueue strategies now that `include` globs are ANDed with the default same-hostname strategy. - The ignore-ssl test now configures TLS verification on the http client, because the crawler-level `ignoreSslErrors` option is not wired to the default client in v4. That dangling option deserves a separate fix or removal, since it currently does nothing. - The impit test pins session fingerprints, since the random default fingerprint overrides the client's browser impersonation. - Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the ES lib and broke compilation on `ErrorOptions`). - Skipped the zero-concurrency queue test: it stages a stuck queue through the v3 client-side `inProgress` set, which the rewritten queue doesn't have. - Fixed the camoufox fetch retry loop fetching 5x even on success, and removed a duplicate `apify` dependency key that silently downgraded the curl-impersonate actor to SDK v3. - Commented out the LOCAL storage matrix entry in the workflow, as `@apify/storage-local` doesn't support v4.
- closes #3998 Unnamed storages (default and aliased) are now purged on start, named ones are not — crawlee-python's rule. The fs backend sweeps the storage directories, so leftovers from a previous process are caught too. Two adjacent bugs, one commit each: - `createDatasetBackend()` / `({})` didn't open the default storage, though `StorageIdentifier` says they do - the `__default__` sentinel leaked into the directory name, so default storages lived in `storage/datasets/__default__` rather than `default` `ThrottlingRequestManager` (#3741) sub-queues are alias-keyed, so they now only survive a restart with `purgeOnStart` off — as its docs already said.
Gets the E2E test suite running against v4. The suite hadn't been run since the v4 rewrite and everything failed on startup. After these changes the MEMORY run passes locally end to end, and getting there surfaced a few real regressions in the packages themselves. - `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global `document` (which doesn't exist in Node) instead of the parsed window, so every call crashed at runtime. - `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed and `firstErrorHtmlUrl` never made it into the crawler statistics. - `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override `enqueueLinks`, exposing the strict urls-required signature even though the runtime helper extracts URLs from the parsed document. - `LinkeDOMCrawler` can now be constructed without arguments, like the other crawlers. - Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports `snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists there). - Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless key-value records; the short-lived `__default__` directory alias it originally targeted was a bug, fixed in #4013) and to the `@crawlee/utils` exports split. - Migrated test actors to the v4 APIs: the `logger` option with `ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the crawling context, `session.setCookie()`, a custom `SessionPool` instead of `sessionPoolOptions`, the WHATWG `Response` returned by `sendRequest`, `registerDeferredCleanup` for dataset writes that must survive a throwing handler, and explicit enqueue strategies now that `include` globs are ANDed with the default same-hostname strategy. - The ignore-ssl test now configures TLS verification on the http client, because the crawler-level `ignoreSslErrors` option is not wired to the default client in v4. That dangling option deserves a separate fix or removal, since it currently does nothing. - The impit test pins session fingerprints, since the random default fingerprint overrides the client's browser impersonation. - Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the ES lib and broke compilation on `ErrorOptions`). - Skipped the zero-concurrency queue test: it stages a stuck queue through the v3 client-side `inProgress` set, which the rewritten queue doesn't have. - Fixed the camoufox fetch retry loop fetching 5x even on success, and removed a duplicate `apify` dependency key that silently downgraded the curl-impersonate actor to SDK v3. - Commented out the LOCAL storage matrix entry in the workflow, as `@apify/storage-local` doesn't support v4.
- closes #3998 Unnamed storages (default and aliased) are now purged on start, named ones are not — crawlee-python's rule. The fs backend sweeps the storage directories, so leftovers from a previous process are caught too. Two adjacent bugs, one commit each: - `createDatasetBackend()` / `({})` didn't open the default storage, though `StorageIdentifier` says they do - the `__default__` sentinel leaked into the directory name, so default storages lived in `storage/datasets/__default__` rather than `default` `ThrottlingRequestManager` (#3741) sub-queues are alias-keyed, so they now only survive a restart with `purgeOnStart` off — as its docs already said.
Gets the E2E test suite running against v4. The suite hadn't been run since the v4 rewrite and everything failed on startup. After these changes the MEMORY run passes locally end to end, and getting there surfaced a few real regressions in the packages themselves. - `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global `document` (which doesn't exist in Node) instead of the parsed window, so every call crashed at runtime. - `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed and `firstErrorHtmlUrl` never made it into the crawler statistics. - `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override `enqueueLinks`, exposing the strict urls-required signature even though the runtime helper extracts URLs from the parsed document. - `LinkeDOMCrawler` can now be constructed without arguments, like the other crawlers. - Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports `snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists there). - Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless key-value records; the short-lived `__default__` directory alias it originally targeted was a bug, fixed in #4013) and to the `@crawlee/utils` exports split. - Migrated test actors to the v4 APIs: the `logger` option with `ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the crawling context, `session.setCookie()`, a custom `SessionPool` instead of `sessionPoolOptions`, the WHATWG `Response` returned by `sendRequest`, `registerDeferredCleanup` for dataset writes that must survive a throwing handler, and explicit enqueue strategies now that `include` globs are ANDed with the default same-hostname strategy. - The ignore-ssl test now configures TLS verification on the http client, because the crawler-level `ignoreSslErrors` option is not wired to the default client in v4. That dangling option deserves a separate fix or removal, since it currently does nothing. - The impit test pins session fingerprints, since the random default fingerprint overrides the client's browser impersonation. - Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the ES lib and broke compilation on `ErrorOptions`). - Skipped the zero-concurrency queue test: it stages a stuck queue through the v3 client-side `inProgress` set, which the rewritten queue doesn't have. - Fixed the camoufox fetch retry loop fetching 5x even on success, and removed a duplicate `apify` dependency key that silently downgraded the curl-impersonate actor to SDK v3. - Commented out the LOCAL storage matrix entry in the workflow, as `@apify/storage-local` doesn't support v4.
purgeOnStart#3998Unnamed storages (default and aliased) are now purged on start, named ones are not — crawlee-python's rule. The fs backend sweeps the storage directories, so leftovers from a previous process are caught too.
Two adjacent bugs, one commit each:
createDatasetBackend()/({})didn't open the default storage, thoughStorageIdentifiersays they do__default__sentinel leaked into the directory name, so default storages lived instorage/datasets/__default__rather thandefaultThrottlingRequestManager(#3741) sub-queues are alias-keyed, so they now only survive a restart withpurgeOnStartoff — as its docs already said.