diff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdx
index c51281dfe30b..5fd8ea33dce6 100644
--- a/docs/guides/configuration.mdx
+++ b/docs/guides/configuration.mdx
@@ -90,7 +90,7 @@ The default request queue has ID `default`. Setting this environment variable ov
#### `CRAWLEE_PURGE_ON_START`
-Storage directories are purged by default. If set to `false` - local storage directories would not be purged automatically at the start of the crawler run or before opening of some storage explicitly (e.g. via `Dataset.open()`). Useful if we're trying e.g. to add more items to dataset with each next run (and keep the previously saved/scraped items).
+Run-scoped storage directories (the default storage and any opened with an `alias`) are purged by default; named storages are never purged. If set to `false` - local storage directories would not be purged automatically at the start of the crawler run or before opening of some storage explicitly (e.g. via `Dataset.open()`). Useful if we're trying e.g. to add more items to dataset with each next run (and keep the previously saved/scraped items).
#### `CRAWLEE_CONTAINERIZED`
diff --git a/docs/guides/request_loaders.mdx b/docs/guides/request_loaders.mdx
index 949643c7416e..a21283ae9f22 100644
--- a/docs/guides/request_loaders.mdx
+++ b/docs/guides/request_loaders.mdx
@@ -160,7 +160,7 @@ const crawler = new CheerioCrawler({
Requests for a listed domain are routed into their own queue as they are added. When one of those domains answers with a 429, the crawler honours its `Retry-After` header — or backs off exponentially from `baseDelaySecs` up to `maxDelaySecs` if there is none — and holds that domain's requests back for the duration. Requests for every other domain keep flowing at full speed, the throttled request is retried later without counting against `maxRequestRetries`, and its session is left alone, because a rate limit says nothing about the session.
-Because a throttled request costs no retries, a domain that never stops rate-limiting would otherwise keep the crawl alive forever. If one goes `maxDomainStallSecs` without letting a single request through, the crawl shuts down with a `PersistentRateLimitError` — at that point the concurrency is too high for that domain, or it has blocked you outright, and waiting longer will not help. Its requests are left in their queue on purpose, so re-running the crawl without purging storages resumes them if the rate limit lifts. A crawler running with `keepAlive` is exempt, since staying up regardless is what it was asked to do.
+Because a throttled request costs no retries, a domain that never stops rate-limiting would otherwise keep the crawl alive forever. If one goes `maxDomainStallSecs` without letting a single request through, the crawl shuts down with a `PersistentRateLimitError` — at that point the concurrency is too high for that domain, or it has blocked you outright, and waiting longer will not help. Its requests are left in their queue on purpose, so re-running the crawl with `purgeOnStart` disabled resumes them if the rate limit lifts. A crawler running with `keepAlive` is exempt, since staying up regardless is what it was asked to do.
This is opt-in and exact: only the domains you list are throttled, and matching is case-insensitive with no wildcard support, so list each subdomain you care about.
diff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdx
index 42c49ef37356..01f1808efd2c 100644
--- a/docs/guides/request_storage.mdx
+++ b/docs/guides/request_storage.mdx
@@ -80,7 +80,7 @@ See the dedicated [Request loaders](./request-loaders) guide for details on load
## Cleaning up the storages
-Default storages are purged before the crawler starts if not specified otherwise. This happens as early as when we try to open some storage (e.g. via `RequestQueue.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `crawler.addRequests()` that under the hood calls `RequestQueue.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the `purgeDefaultStorages()` helper explicitly:
+Run-scoped storages - the default one and any opened with an `alias` - are purged before the crawler starts if not specified otherwise. Storages opened with a `name` persist across runs and are never purged. This happens as early as when we try to open some storage (e.g. via `RequestQueue.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `crawler.addRequests()` that under the hood calls `RequestQueue.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the `purgeDefaultStorages()` helper explicitly:
```javascript
import { purgeDefaultStorages } from 'crawlee';
@@ -88,4 +88,4 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default request storage directory (and also the request list stored in default key-value store). This is a shortcut for running (optional) `purge` method on the `StorageBackend` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
+Calling this function will clean up the run-scoped request storage directories - the default queue and any alias-keyed one - along with the request list stored in the default key-value store. This is a shortcut for running (optional) `purge` method on the `StorageBackend` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
diff --git a/docs/guides/result_storage.mdx b/docs/guides/result_storage.mdx
index b2354166bd51..af17763e0fbc 100644
--- a/docs/guides/result_storage.mdx
+++ b/docs/guides/result_storage.mdx
@@ -102,7 +102,7 @@ To see how to use the dataset to store crawler results, see the [Cheerio Crawler
## Cleaning up the storages
-Default storages are purged before the crawler starts if not specified otherwise. This happens as early as when we try to open some storage (e.g. via `Dataset.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `Dataset.pushData()` that under the hood calls `Dataset.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the `purgeDefaultStorages()` helper explicitly:
+Run-scoped storages - the default one and any opened with an `alias` - are purged before the crawler starts if not specified otherwise. Storages opened with a `name` persist across runs and are never purged. This happens as early as when we try to open some storage (e.g. via `Dataset.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `Dataset.pushData()` that under the hood calls `Dataset.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the `purgeDefaultStorages()` helper explicitly:
```javascript
import { purgeDefaultStorages } from 'crawlee';
@@ -110,7 +110,7 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the `StorageBackend` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
+Calling this function will clean up the run-scoped results storage directories - the default ones and any alias-keyed one - except the `INPUT` key in the default key-value store directory. This is a shortcut for running (optional) `purge` method on the `StorageBackend` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
## Transactional storage
diff --git a/docs/public-api/crawlee-types.api.md b/docs/public-api/crawlee-types.api.md
index 87cac4e4b1ca..addfeff191ac 100644
--- a/docs/public-api/crawlee-types.api.md
+++ b/docs/public-api/crawlee-types.api.md
@@ -570,7 +570,6 @@ export interface StorageBackend {
createKeyValueStoreBackend(options?: StorageIdentifier): Promise;
createRequestQueueBackend(options?: StorageIdentifier): Promise;
getStorageBackendCacheKey?(): string;
- // (undocumented)
purge?(): Promise;
// (undocumented)
stats?: {
diff --git a/packages/core/src/memory-storage/memory-storage.ts b/packages/core/src/memory-storage/memory-storage.ts
index 77f218ed61b0..1cdd751a23ba 100644
--- a/packages/core/src/memory-storage/memory-storage.ts
+++ b/packages/core/src/memory-storage/memory-storage.ts
@@ -7,6 +7,9 @@ import { DatasetBackend } from './resource-clients/dataset.js';
import { KeyValueStoreBackend } from './resource-clients/key-value-store.js';
import { RequestQueueBackend } from './resource-clients/request-queue.js';
+/** The alias the default (unnamed) storage is opened under. */
+const DEFAULT_STORAGE_ALIAS = '__default__';
+
export interface MemoryStorageOptions {
/**
* Optional logger for MemoryStorageBackend warnings.
@@ -39,30 +42,32 @@ export class MemoryStorageBackend implements storage.StorageBackend {
return this.#instanceCacheKey;
}
- private static resolveStorageKey(options: { id?: string; name?: string; alias?: string }): {
+ static #resolveStorageKey(options: { id?: string; name?: string; alias?: string }): {
isAlias: boolean;
- cacheKey: string | undefined;
+ cacheKey: string;
} {
- const isAlias = 'alias' in options && !!options.alias;
- const rawKey = isAlias ? options.alias : (options.name ?? options.id);
+ // No identifier at all means the default storage, which is opened under the reserved alias —
+ // same rule as `resolveStorageIdentifier` in the storage frontends, so that a backend used
+ // directly lands on the very storage the frontends would have opened.
+ const alias = options.alias || (!options.id && !options.name ? DEFAULT_STORAGE_ALIAS : undefined);
+ // `alias` covers the identifier-less case, so one of the three is always set.
+ const rawKey = alias ?? options.name ?? options.id!;
// Normalize the internal __default__ alias to the user-facing 'default' name.
- const cacheKey = rawKey === '__default__' ? 'default' : rawKey;
- return { isAlias, cacheKey };
+ const cacheKey = rawKey === DEFAULT_STORAGE_ALIAS ? 'default' : rawKey;
+ return { isAlias: alias !== undefined, cacheKey };
}
async createDatasetBackend(options: storage.StorageIdentifier = {}): Promise {
- const { isAlias, cacheKey } = MemoryStorageBackend.resolveStorageKey(options);
-
- if (cacheKey) {
- const found = this.datasetBackendCache.find(
- (store) =>
- store.id === cacheKey ||
- store.name?.toLowerCase() === cacheKey.toLowerCase() ||
- store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
- );
- if (found) {
- return found;
- }
+ const { isAlias, cacheKey } = MemoryStorageBackend.#resolveStorageKey(options);
+
+ const found = this.datasetBackendCache.find(
+ (store) =>
+ store.id === cacheKey ||
+ store.name?.toLowerCase() === cacheKey.toLowerCase() ||
+ store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
+ );
+ if (found) {
+ return found;
}
const newStore = new DatasetBackend({
@@ -76,18 +81,16 @@ export class MemoryStorageBackend implements storage.StorageBackend {
}
async createKeyValueStoreBackend(options: storage.StorageIdentifier = {}): Promise {
- const { isAlias, cacheKey } = MemoryStorageBackend.resolveStorageKey(options);
-
- if (cacheKey) {
- const found = this.keyValueStoreBackendCache.find(
- (store) =>
- store.id === cacheKey ||
- store.name?.toLowerCase() === cacheKey.toLowerCase() ||
- store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
- );
- if (found) {
- return found;
- }
+ const { isAlias, cacheKey } = MemoryStorageBackend.#resolveStorageKey(options);
+
+ const found = this.keyValueStoreBackendCache.find(
+ (store) =>
+ store.id === cacheKey ||
+ store.name?.toLowerCase() === cacheKey.toLowerCase() ||
+ store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
+ );
+ if (found) {
+ return found;
}
const newStore = new KeyValueStoreBackend({
@@ -101,18 +104,16 @@ export class MemoryStorageBackend implements storage.StorageBackend {
}
async createRequestQueueBackend(options: storage.StorageIdentifier = {}): Promise {
- const { isAlias, cacheKey } = MemoryStorageBackend.resolveStorageKey(options);
-
- if (cacheKey) {
- const found = this.requestQueueBackendCache.find(
- (queue) =>
- queue.id === cacheKey ||
- queue.name?.toLowerCase() === cacheKey.toLowerCase() ||
- queue.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
- );
- if (found) {
- return found;
- }
+ const { isAlias, cacheKey } = MemoryStorageBackend.#resolveStorageKey(options);
+
+ const found = this.requestQueueBackendCache.find(
+ (queue) =>
+ queue.id === cacheKey ||
+ queue.name?.toLowerCase() === cacheKey.toLowerCase() ||
+ queue.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
+ );
+ if (found) {
+ return found;
}
const newStore = new RequestQueueBackend({
@@ -147,34 +148,31 @@ export class MemoryStorageBackend implements storage.StorageBackend {
}
/**
- * Cleans up the default storages before the run starts. For the in-memory storage this simply
- * resets the in-memory state of the cached default dataset, key-value store and request queue.
- *
- * As with `FileSystemStorageBackend`, the run's input (the `INPUT` key in the default key-value
- * store) is preserved — only the rest of the default storages is cleared.
+ * Cleans up the run-scoped storages before the run starts. For the in-memory storage this simply
+ * resets the in-memory state of the cached backends.
*/
async purge(): Promise {
- // The run default is opened via `{ alias: '__default__' }`, which `resolveStorageKey`
- // normalizes to `cacheKey === 'default'` (with `name === undefined`) — that is the clause
- // that actually matches it. The `name === 'default'` clause additionally covers a store a user
- // explicitly opened via `{ name: 'default' }`. (`'__default__'` never reaches `cacheKey`,
- // as it is always normalized to `'default'` first, so it does not need to be checked here.)
+ // `#resolveStorageKey` leaves `name` unset for the default and alias-keyed storages, which is what
+ // marks them as run-scoped. `'default'` is the exception — it collapses onto the default storage.
+ const isRunScoped = (store: { name?: string }) => store.name === undefined || store.name === 'default';
+
const isDefault = (store: { name?: string; cacheKey: string }) =>
store.name === 'default' || store.cacheKey === 'default';
- const purgeDefaults = async (
+ const purgeRunScoped = async (
cache: T[],
purgeStore: (store: T) => Promise,
) => {
- await Promise.all(cache.filter(isDefault).map(async (store) => purgeStore(store)));
+ await Promise.all(cache.filter(isRunScoped).map(async (store) => purgeStore(store)));
};
await Promise.all([
- // Preserve the run input (INPUT) when purging the default key-value store, matching
- // `FileSystemStorageBackend`.
- purgeDefaults(this.keyValueStoreBackendCache, async (store) => store.purgeExceptInput()),
- purgeDefaults(this.datasetBackendCache, async (store) => store.purge()),
- purgeDefaults(this.requestQueueBackendCache, async (store) => store.purge()),
+ // Only the default store holds the run input, so it is the only one that keeps `INPUT`.
+ purgeRunScoped(this.keyValueStoreBackendCache, async (store) =>
+ isDefault(store) ? store.purgeExceptInput() : store.purge(),
+ ),
+ purgeRunScoped(this.datasetBackendCache, async (store) => store.purge()),
+ purgeRunScoped(this.requestQueueBackendCache, async (store) => store.purge()),
]);
}
diff --git a/packages/core/src/storages/throttling_request_manager.ts b/packages/core/src/storages/throttling_request_manager.ts
index 6db5f7efc206..c3ac8a739c72 100644
--- a/packages/core/src/storages/throttling_request_manager.ts
+++ b/packages/core/src/storages/throttling_request_manager.ts
@@ -101,7 +101,7 @@ export interface ThrottlingRequestManagerOptions;
diff --git a/packages/core/src/storages/utils.ts b/packages/core/src/storages/utils.ts
index 1b3b2a16c837..5bbbc01ba003 100644
--- a/packages/core/src/storages/utils.ts
+++ b/packages/core/src/storages/utils.ts
@@ -21,7 +21,8 @@ interface PurgeDefaultStorageOptions {
/**
* Cleans up the local storage folder (defaults to `./storage`) created when running code locally.
- * Purging will remove all the files in all storages except for INPUT.json in the default KV store.
+ * Purging empties the storages that belong to a single run — the default one and every alias-keyed one —
+ * keeping only INPUT.json in the default KV store. Named storages persist across runs and are not touched.
*
* Purging of storages is happening automatically when we run our crawler (or when we open some storage
* explicitly, e.g. via `RequestList.open()`). We can disable that via `purgeOnStart` {@apilink Configuration}
@@ -35,7 +36,8 @@ interface PurgeDefaultStorageOptions {
export async function purgeDefaultStorages(options?: PurgeDefaultStorageOptions): Promise;
/**
* Cleans up the local storage folder (defaults to `./storage`) created when running code locally.
- * Purging will remove all the files in all storages except for INPUT.json in the default KV store.
+ * Purging empties the storages that belong to a single run — the default one and every alias-keyed one —
+ * keeping only INPUT.json in the default KV store. Named storages persist across runs and are not touched.
*
* Purging of storages is happening automatically when we run our crawler (or when we open some storage
* explicitly, e.g. via `RequestList.open()`). We can disable that via `purgeOnStart` {@apilink Configuration}
diff --git a/packages/fs-storage/src/file-system-storage.ts b/packages/fs-storage/src/file-system-storage.ts
index eb68969c7687..656961a10a5a 100644
--- a/packages/fs-storage/src/file-system-storage.ts
+++ b/packages/fs-storage/src/file-system-storage.ts
@@ -14,6 +14,12 @@ import { DatasetBackend } from './resource-clients/dataset.js';
import { KeyValueStoreBackend } from './resource-clients/key-value-store.js';
import { RequestQueueBackend } from './resource-clients/request-queue.js';
+/** The alias `@crawlee/core` opens the default (unnamed) storage under. */
+const DEFAULT_STORAGE_ALIAS = '__default__';
+
+/** The directory the default storage lives in, one level below `datasets` / `key_value_stores` / etc. */
+const DEFAULT_STORAGE_DIRECTORY = 'default';
+
export interface FileSystemStorageOptions {
/**
* Path to directory where the data will be saved.
@@ -87,38 +93,42 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
return `FileSystemStorageBackend:${resolve(this.localDataDirectory)}`;
}
- private static resolveStorageKey(options: { id?: string; name?: string; alias?: string }): {
+ static #resolveStorageKey(options: { id?: string; name?: string; alias?: string }): {
id?: string;
name?: string;
alias?: string;
- cacheKey: string | undefined;
+ cacheKey: string;
} {
- const isAlias = 'alias' in options && !!options.alias;
- const rawKey = isAlias ? options.alias : (options.name ?? options.id);
- // Normalize the internal __default__ alias to the user-facing 'default' name.
- const cacheKey = rawKey === '__default__' ? 'default' : rawKey;
- return { id: options.id, name: options.name, alias: options.alias, cacheKey };
+ // No identifier at all means the default storage, which is opened under the reserved alias —
+ // same rule as `resolveStorageIdentifier` in @crawlee/core, so that a backend used directly
+ // lands on the very storage the frontends would have opened.
+ const requestedAlias = options.alias || (!options.id && !options.name ? DEFAULT_STORAGE_ALIAS : undefined);
+ // `__default__` is an internal sentinel and must not escape onto disk: the default storage lives
+ // in `default`, which is what the docs, the project templates and every pre-existing local
+ // `storage/` directory expect. Normalizing here keeps the cache key and the directory in step.
+ const alias = requestedAlias === DEFAULT_STORAGE_ALIAS ? DEFAULT_STORAGE_DIRECTORY : requestedAlias;
+ // `alias` covers the identifier-less case, so one of the three is always set.
+ const cacheKey = alias ?? options.name ?? options.id!;
+ return { id: options.id, name: options.name, alias, cacheKey };
}
async createDatasetBackend(options: storage.StorageIdentifier = {}): Promise {
- const { id, name, alias, cacheKey } = FileSystemStorageBackend.resolveStorageKey(options);
-
- if (cacheKey) {
- const found = this.datasetBackendCache.find(
- (store) =>
- store.id === cacheKey ||
- store.name?.toLowerCase() === cacheKey.toLowerCase() ||
- store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
- );
- if (found) {
- return found;
- }
+ const { id, name, alias, cacheKey } = FileSystemStorageBackend.#resolveStorageKey(options);
+
+ const found = this.datasetBackendCache.find(
+ (store) =>
+ store.id === cacheKey ||
+ store.name?.toLowerCase() === cacheKey.toLowerCase() ||
+ store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
+ );
+ if (found) {
+ return found;
}
const nativeBackend = await NativeDatasetBackend.open(id, name, alias, this.localDataDirectory);
const newStore = await DatasetBackend.create({
name: alias ? undefined : (name ?? cacheKey),
- cacheKey: cacheKey ?? '',
+ cacheKey,
nativeBackend,
logger: this.logger,
});
@@ -128,24 +138,22 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
}
async createKeyValueStoreBackend(options: storage.StorageIdentifier = {}): Promise {
- const { id, name, alias, cacheKey } = FileSystemStorageBackend.resolveStorageKey(options);
-
- if (cacheKey) {
- const found = this.keyValueStoreBackendCache.find(
- (store) =>
- store.id === cacheKey ||
- store.name?.toLowerCase() === cacheKey.toLowerCase() ||
- store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
- );
- if (found) {
- return found;
- }
+ const { id, name, alias, cacheKey } = FileSystemStorageBackend.#resolveStorageKey(options);
+
+ const found = this.keyValueStoreBackendCache.find(
+ (store) =>
+ store.id === cacheKey ||
+ store.name?.toLowerCase() === cacheKey.toLowerCase() ||
+ store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
+ );
+ if (found) {
+ return found;
}
const nativeBackend = await NativeKeyValueStoreBackend.open(id, name, alias, this.localDataDirectory);
const newStore = await KeyValueStoreBackend.create({
name: alias ? undefined : (name ?? cacheKey),
- cacheKey: cacheKey ?? '',
+ cacheKey,
nativeBackend,
logger: this.logger,
});
@@ -155,18 +163,16 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
}
async createRequestQueueBackend(options: storage.StorageIdentifier = {}): Promise {
- const { id, name, alias, cacheKey } = FileSystemStorageBackend.resolveStorageKey(options);
-
- if (cacheKey) {
- const found = this.requestQueueBackendCache.find(
- (queue) =>
- queue.id === cacheKey ||
- queue.name?.toLowerCase() === cacheKey.toLowerCase() ||
- queue.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
- );
- if (found) {
- return found;
- }
+ const { id, name, alias, cacheKey } = FileSystemStorageBackend.#resolveStorageKey(options);
+
+ const found = this.requestQueueBackendCache.find(
+ (queue) =>
+ queue.id === cacheKey ||
+ queue.name?.toLowerCase() === cacheKey.toLowerCase() ||
+ queue.cacheKey.toLowerCase() === cacheKey.toLowerCase(),
+ );
+ if (found) {
+ return found;
}
const nativeBackend = await NativeRequestQueueBackend.open(
@@ -180,7 +186,7 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
);
const newStore = await RequestQueueBackend.create({
name: alias ? undefined : (name ?? cacheKey),
- cacheKey: cacheKey ?? '',
+ cacheKey,
nativeBackend,
logger: this.logger,
});
@@ -224,7 +230,7 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
// has a matching directory. We therefore read the real id from the metadata and only report
// existence when it equals the queried string. This matches upstream PR #3800/#3808 and
// prevents a named storage from being re-resolved as `{ id: name }` on a subsequent run.
- const resolvedId = await FileSystemStorageBackend.resolveStorageIdOnDisk(baseDir, id);
+ const resolvedId = await FileSystemStorageBackend.#resolveStorageIdOnDisk(baseDir, id);
return resolvedId === id;
}
@@ -236,13 +242,10 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
* back to scanning sibling directories for one whose metadata id equals `entryNameOrId` (the case
* of a storage opened by name and later looked up by its auto-assigned id).
*/
- private static async resolveStorageIdOnDisk(
- baseDirectory: string,
- entryNameOrId: string,
- ): Promise {
+ static async #resolveStorageIdOnDisk(baseDirectory: string, entryNameOrId: string): Promise {
// Directory named exactly after the string: return its real (metadata) id, which may differ
// from the string when the string is a name rather than an id.
- const directId = await FileSystemStorageBackend.readMetadataId(resolve(baseDirectory, entryNameOrId));
+ const directId = (await FileSystemStorageBackend.#readMetadata(resolve(baseDirectory, entryNameOrId)))?.id;
if (directId !== undefined) {
return directId;
}
@@ -260,7 +263,8 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
continue;
}
- const metadataId = await FileSystemStorageBackend.readMetadataId(resolve(baseDirectory, directory.name));
+ const metadataId = (await FileSystemStorageBackend.#readMetadata(resolve(baseDirectory, directory.name)))
+ ?.id;
if (metadataId === entryNameOrId) {
return metadataId;
}
@@ -269,44 +273,108 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
return undefined;
}
- /** Read the `id` field from a storage directory's `__metadata__.json`, or `undefined` if absent. */
- private static async readMetadataId(storageDirectory: string): Promise {
+ /** Read a storage directory's `__metadata__.json`, or `undefined` if there is none to read. */
+ static async #readMetadata(storageDirectory: string): Promise<{ id?: string; name?: string | null } | undefined> {
try {
const fileContent = await readFile(resolve(storageDirectory, '__metadata__.json'), 'utf8');
- return (JSON.parse(fileContent) as { id?: string }).id;
+ return JSON.parse(fileContent) as { id?: string; name?: string | null };
} catch {
- // Directory missing, or no/unreadable metadata file — no id to report.
+ // Directory missing, or no/unreadable metadata file — nothing to report.
return undefined;
}
}
/**
- * Cleans up the default storages before the run starts:
- * - the default dataset;
- * - all records from the default key-value store, except for the "INPUT" key;
- * - the default request queue.
+ * Cleans up the run-scoped storages before the run starts, sweeping the storage directories so that
+ * leftovers from a previous process are caught too.
*/
async purge(): Promise {
- // Resolve the default stores up front so leftover on-disk records are purged even when the
- // store has not been opened in this process yet (e.g. a fresh run over a pre-existing
- // directory). Opening caches the backend, so the subsequent purge operates on a real backend.
- // The default store is opened via the internal `__default__` alias (see resolveStorageIdentifier
- // in @crawlee/core), which resolves to the `default` cache key — match that here so we purge the
- // very backend the default open would return rather than creating a divergent one.
- const [defaultKeyValueStore, defaultDataset, defaultRequestQueue] = await Promise.all([
- this.createKeyValueStoreBackend({ alias: '__default__' }) as Promise,
- this.createDatasetBackend({ alias: '__default__' }) as Promise,
- this.createRequestQueueBackend({ alias: '__default__' }) as Promise,
- ]);
-
await Promise.all([
- // Preserve the run input (INPUT) when purging the default key-value store.
- defaultKeyValueStore.purgeExceptInput(),
- defaultDataset.purge(),
- defaultRequestQueue.purge(),
+ this.#purgeRunScopedStorages(
+ this.keyValueStoresDirectory,
+ async (alias) => this.createKeyValueStoreBackend({ alias }) as Promise,
+ // Only the default store holds the run input, so it is the only one that keeps `INPUT`.
+ async (store, isDefault) => (isDefault ? store.purgeExceptInput() : store.purge()),
+ ),
+ this.#purgeRunScopedStorages(
+ this.datasetsDirectory,
+ async (alias) => this.createDatasetBackend({ alias }) as Promise,
+ async (store) => store.purge(),
+ ),
+ this.#purgeRunScopedStorages(
+ this.requestQueuesDirectory,
+ async (alias) => this.createRequestQueueBackend({ alias }) as Promise,
+ async (store) => store.purge(),
+ ),
]);
}
+ /**
+ * Purge every run-scoped storage under `storagesDirectory`, whether or not it has been opened in this
+ * process yet. Storages are opened rather than emptied on disk directly, so that one already open
+ * under the same name or alias is purged through the backend the run is using, not a second one.
+ */
+ async #purgeRunScopedStorages(
+ storagesDirectory: string,
+ open: (alias: string) => Promise,
+ purgeStorage: (storage: T, isDefault: boolean) => Promise,
+ ): Promise {
+ // The default storage is listed unconditionally, so that a run over an empty directory still ends
+ // up with it opened (and cached) exactly as it was before. Deduplicating by cache key then keeps
+ // it to a single open: every run after the first also finds its `default` directory on disk, and
+ // opening the same storage twice concurrently would race two backends onto one directory.
+ const aliasesByCacheKey = new Map();
+
+ for (const alias of [
+ DEFAULT_STORAGE_ALIAS,
+ ...(await FileSystemStorageBackend.#listUnnamedStorages(storagesDirectory)),
+ ]) {
+ const { cacheKey } = FileSystemStorageBackend.#resolveStorageKey({ alias });
+ if (!aliasesByCacheKey.has(cacheKey)) {
+ aliasesByCacheKey.set(cacheKey, alias);
+ }
+ }
+
+ await Promise.all(
+ Array.from(aliasesByCacheKey, async ([cacheKey, alias]) => {
+ await purgeStorage(await open(alias), cacheKey === DEFAULT_STORAGE_DIRECTORY);
+ }),
+ );
+ }
+
+ /**
+ * The directory names of the on-disk storages under `storagesDirectory` that Crawlee created without
+ * a name — the default storage and every alias-keyed one. Since the directory is named after the
+ * storage's `name ?? alias`, the name is read from the metadata rather than guessed.
+ *
+ * Two kinds of directory are left out, as purging them would destroy data this process never wrote:
+ * one without a readable `__metadata__.json` (not written by Crawlee — a hand-placed input directory,
+ * say), and one named after its own id, which is reachable only by `{ id }` and so is not run-scoped.
+ */
+ static async #listUnnamedStorages(storagesDirectory: string): Promise {
+ let directories;
+ try {
+ directories = await opendir(storagesDirectory);
+ } catch {
+ return [];
+ }
+
+ const unnamed: string[] = [];
+
+ for await (const directory of directories) {
+ if (!directory.isDirectory()) {
+ continue;
+ }
+
+ const metadata = await FileSystemStorageBackend.#readMetadata(resolve(storagesDirectory, directory.name));
+ if (metadata !== undefined && typeof metadata.name !== 'string' && metadata.id !== directory.name) {
+ unnamed.push(directory.name);
+ }
+ }
+
+ return unnamed;
+ }
+
/**
* This method should be called at the end of the process, to ensure all data is saved.
*
diff --git a/packages/fs-storage/test/default-storage-layout.test.ts b/packages/fs-storage/test/default-storage-layout.test.ts
new file mode 100644
index 000000000000..46dd50f4c423
--- /dev/null
+++ b/packages/fs-storage/test/default-storage-layout.test.ts
@@ -0,0 +1,55 @@
+import { mkdir, readdir, rm, writeFile } from 'node:fs/promises';
+import { resolve } from 'node:path';
+
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+
+// The default storage lives in `default`. The alias @crawlee/core opens it under is an internal
+// sentinel, and letting that reach the disk orphans every `storage/` directory an earlier run wrote.
+describe('the default storage on disk', () => {
+ const tmpLocation = resolve(import.meta.dirname, './tmp/default-storage-layout');
+
+ afterEach(async () => {
+ await rm(tmpLocation, { force: true, recursive: true });
+ });
+
+ test('every default storage lands in a `default` directory', async () => {
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
+
+ await storage.createDatasetBackend();
+ await storage.createKeyValueStoreBackend();
+ await storage.createRequestQueueBackend();
+
+ expect(await readdir(storage.datasetsDirectory)).toEqual(['default']);
+ expect(await readdir(storage.keyValueStoresDirectory)).toEqual(['default']);
+ expect(await readdir(storage.requestQueuesDirectory)).toEqual(['default']);
+ });
+
+ // The documented way to supply input to a local run: drop a file into the default key-value store
+ // directory by hand. It only works if that directory is the one the default store actually opens.
+ test('reads an INPUT.json placed in the default key-value store directory by hand', async () => {
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
+ await mkdir(resolve(storage.keyValueStoresDirectory, 'default'), { recursive: true });
+ await writeFile(
+ resolve(storage.keyValueStoresDirectory, 'default', 'INPUT.json'),
+ JSON.stringify({ hello: 'world' }),
+ );
+
+ const defaultStore = await storage.createKeyValueStoreBackend();
+
+ expect((await defaultStore.getValue('INPUT'))?.value.toString()).toBe(JSON.stringify({ hello: 'world' }));
+ });
+
+ test('keeps a hand-placed INPUT.json across a purge', async () => {
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
+ await mkdir(resolve(storage.keyValueStoresDirectory, 'default'), { recursive: true });
+ await writeFile(
+ resolve(storage.keyValueStoresDirectory, 'default', 'INPUT.json'),
+ JSON.stringify({ hello: 'world' }),
+ );
+
+ await storage.purge();
+
+ const defaultStore = await storage.createKeyValueStoreBackend();
+ expect((await defaultStore.getValue('INPUT'))?.value.toString()).toBe(JSON.stringify({ hello: 'world' }));
+ });
+});
diff --git a/packages/types/src/storages.ts b/packages/types/src/storages.ts
index f54792ed7fe4..f8453cf4ac38 100644
--- a/packages/types/src/storages.ts
+++ b/packages/types/src/storages.ts
@@ -392,10 +392,12 @@ export interface RequestQueueBackend {
* Identifies a storage by its ID, name, or alias. At most one may be provided.
*
* - `{ id }` — open a pre-existing storage by its unique ID.
- * - `{ name }` — open or create a globally named storage (persists across runs).
+ * - `{ name }` — open or create a globally named storage (persists across runs). The name `default`
+ * is reserved: it resolves to the default storage, and is emptied on start along with it.
* - `{ alias }` — open or create a run-scoped unnamed storage identified by this alias.
* The alias is used locally (e.g. as a directory name or cache key) but the storage
- * itself has no persistent name. Use this for non-default unnamed storages.
+ * itself has no persistent name. Use this for non-default unnamed storages. Like the
+ * default storage, an aliased one is emptied on start unless `purgeOnStart` is disabled.
* - `{}` / omitted — open the default storage.
*/
export type StorageIdentifier =
@@ -456,6 +458,11 @@ export interface StorageBackend {
* `StorageBackend` implementations automatically get separate cache partitions.
*/
getStorageBackendCacheKey?(): string;
+
+ /**
+ * Empty the run-scoped storages — the default one and every alias-keyed one, including any left
+ * behind by a previous run. Named storages persist across runs, as does the default store's `INPUT`.
+ */
purge?(): Promise;
teardown?(): Promise;
stats?: { rateLimitErrors: number[] };
diff --git a/test/core/storages/storage_aliases.test.ts b/test/core/storages/storage_aliases.test.ts
index 5ca3a9220284..fdb0f8c37589 100644
--- a/test/core/storages/storage_aliases.test.ts
+++ b/test/core/storages/storage_aliases.test.ts
@@ -2,6 +2,7 @@ import { resolve } from 'node:path';
import { FileSystemStorageBackend } from '@crawlee/fs-storage';
import { Dataset, KeyValueStore, MemoryStorageBackend, RequestQueue, serviceLocator } from '@crawlee/core';
+import type { StorageBackend } from '@crawlee/types';
import { ensureDir, rm } from 'fs-extra';
import { cryptoRandomObjectId } from '@apify/utilities';
@@ -232,6 +233,12 @@ describe('storage aliases', () => {
expect(dataset1).toBe(dataset2);
});
+ test('the reserved __default__ alias opens default storage', async () => {
+ const dataset1 = await Dataset.open({ alias: '__default__' });
+ const dataset2 = await Dataset.open();
+ expect(dataset1).toBe(dataset2);
+ });
+
test('string identifier opens named storage', async () => {
const dataset = await Dataset.open('test-named');
expect(dataset.name).toBe('test-named');
@@ -246,6 +253,32 @@ describe('storage aliases', () => {
const dataset = await Dataset.open({ alias: 'test-alias' });
expect(dataset.name).toBeUndefined();
});
+
+ describe('at the backend level', () => {
+ const localStorageDir = resolve(import.meta.dirname, '..', 'tmp', 'fs-aliases', cryptoRandomObjectId(10));
+
+ afterAll(async () => {
+ await rm(localStorageDir, { force: true, recursive: true });
+ });
+
+ test.each([
+ ['MemoryStorageBackend', (): StorageBackend => new MemoryStorageBackend()],
+ [
+ 'FileSystemStorageBackend',
+ (): StorageBackend => new FileSystemStorageBackend({ localDataDirectory: localStorageDir }),
+ ],
+ ])(
+ '%s opens the same default storage for the reserved alias and for no identifier',
+ async (_backendName, createBackend) => {
+ const backend = createBackend();
+
+ const aliased = await backend.createDatasetBackend({ alias: '__default__' });
+
+ expect(await backend.createDatasetBackend({})).toBe(aliased);
+ expect(await backend.createDatasetBackend()).toBe(aliased);
+ },
+ );
+ });
});
describe('drop with alias', () => {
diff --git a/test/core/storages/storage_purge.test.ts b/test/core/storages/storage_purge.test.ts
new file mode 100644
index 000000000000..d3d69e727b4c
--- /dev/null
+++ b/test/core/storages/storage_purge.test.ts
@@ -0,0 +1,190 @@
+import { mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises';
+import { resolve } from 'node:path';
+
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import { MemoryStorageBackend, RequestQueue, purgeDefaultStorages, serviceLocator } from '@crawlee/core';
+import type { KeyValueStoreBackend, RequestQueueBackend, StorageBackend } from '@crawlee/types';
+
+import { cryptoRandomObjectId } from '@apify/utilities';
+
+const temporaryRoot = resolve(import.meta.dirname, '..', 'tmp', 'storage-purge');
+
+const temporaryDirectory = () => resolve(temporaryRoot, cryptoRandomObjectId(10));
+
+const requestOf = (url: string) => ({ url, uniqueKey: url });
+
+const pendingCount = async (queue: RequestQueueBackend) => (await queue.getMetadata()).pendingRequestCount;
+
+const readInput = async (store: KeyValueStoreBackend) => (await store.getValue('INPUT'))?.value.toString();
+
+const input = { key: 'INPUT', value: '{"run":"input"}', contentType: 'application/json; charset=utf-8' };
+
+afterAll(async () => {
+ await rm(temporaryRoot, { force: true, recursive: true });
+});
+
+// A `purgeOnStart` purge wipes the storages that belong to a single run: the default one and any
+// alias-keyed one. Named storages are the opt-in "keep this across runs" mechanism and must survive.
+describe.each([
+ ['MemoryStorageBackend', (): StorageBackend => new MemoryStorageBackend()],
+ [
+ 'FileSystemStorageBackend',
+ (): StorageBackend => new FileSystemStorageBackend({ localDataDirectory: temporaryDirectory() }),
+ ],
+])('%s.purge', (_backendName, createBackend) => {
+ test('empties alias-keyed storages along with the default one', async () => {
+ const backend = createBackend();
+
+ const defaultQueue = await backend.createRequestQueueBackend();
+ const aliasQueue = await backend.createRequestQueueBackend({ alias: 'run-scoped' });
+ const defaultDataset = await backend.createDatasetBackend();
+ const aliasDataset = await backend.createDatasetBackend({ alias: 'run-scoped' });
+
+ await defaultQueue.addBatchOfRequests([requestOf('https://example.com/default')]);
+ await aliasQueue.addBatchOfRequests([requestOf('https://example.com/alias')]);
+ await defaultDataset.pushData([{ from: 'default' }]);
+ await aliasDataset.pushData([{ from: 'alias' }]);
+
+ await backend.purge!();
+
+ expect(await pendingCount(defaultQueue)).toBe(0);
+ expect(await pendingCount(aliasQueue)).toBe(0);
+ await expect(defaultDataset.getData()).resolves.toMatchObject({ items: [] });
+ await expect(aliasDataset.getData()).resolves.toMatchObject({ items: [] });
+ });
+
+ test('keeps named storages', async () => {
+ const backend = createBackend();
+
+ const namedQueue = await backend.createRequestQueueBackend({ name: 'persistent' });
+ const namedDataset = await backend.createDatasetBackend({ name: 'persistent' });
+
+ await namedQueue.addBatchOfRequests([requestOf('https://example.com/named')]);
+ await namedDataset.pushData([{ from: 'named' }]);
+
+ await backend.purge!();
+
+ expect(await pendingCount(namedQueue)).toBe(1);
+ await expect(namedDataset.getData()).resolves.toMatchObject({ items: [{ from: 'named' }] });
+ });
+
+ // The run input lives in the default key-value store, so that one store keeps its `INPUT` key.
+ // An alias-keyed store is just another run-scoped storage — nothing there is the run input.
+ test('keeps INPUT in the default key-value store but not in an alias-keyed one', async () => {
+ const backend = createBackend();
+
+ const defaultStore = await backend.createKeyValueStoreBackend();
+ const aliasStore = await backend.createKeyValueStoreBackend({ alias: 'run-scoped' });
+
+ await defaultStore.setValue(input);
+ await aliasStore.setValue(input);
+
+ await backend.purge!();
+
+ expect(await readInput(defaultStore)).toBe(input.value);
+ expect(await readInput(aliasStore)).toBeUndefined();
+ });
+});
+
+// The file system backend has to find leftovers on disk, since a fresh process starts with an empty
+// backend cache and knows nothing about the storages the previous run opened.
+describe('FileSystemStorageBackend.purge over a pre-existing storage directory', () => {
+ const localDataDirectory = temporaryDirectory();
+
+ test('empties an alias-keyed queue left behind by a previous process', async () => {
+ const firstRun = new FileSystemStorageBackend({ localDataDirectory });
+ const queue = await firstRun.createRequestQueueBackend({ alias: 'throttled-example.com' });
+ await queue.addBatchOfRequests([requestOf('https://example.com/left-behind')]);
+ await firstRun.teardown();
+
+ const secondRun = new FileSystemStorageBackend({ localDataDirectory });
+ await secondRun.purge();
+
+ const reopened = await secondRun.createRequestQueueBackend({ alias: 'throttled-example.com' });
+ expect(await pendingCount(reopened)).toBe(0);
+ });
+
+ test('keeps a named queue left behind by a previous process', async () => {
+ const firstRun = new FileSystemStorageBackend({ localDataDirectory });
+ const queue = await firstRun.createRequestQueueBackend({ name: 'persistent-across-runs' });
+ await queue.addBatchOfRequests([requestOf('https://example.com/keep-me')]);
+ await firstRun.teardown();
+
+ const secondRun = new FileSystemStorageBackend({ localDataDirectory });
+ await secondRun.purge();
+
+ const reopened = await secondRun.createRequestQueueBackend({ name: 'persistent-across-runs' });
+ expect(await pendingCount(reopened)).toBe(1);
+ });
+
+ // A directory without `__metadata__.json` was not created by Crawlee — most likely a hand-placed
+ // input directory. Purging it would delete data we never wrote.
+ test('leaves a storage directory it did not create alone', async () => {
+ const foreignDirectory = temporaryDirectory();
+ const backend = new FileSystemStorageBackend({ localDataDirectory: foreignDirectory });
+ await mkdir(resolve(backend.keyValueStoresDirectory, 'hand-placed'), { recursive: true });
+ await writeFile(resolve(backend.keyValueStoresDirectory, 'hand-placed', 'INPUT.json'), '{"hand":"placed"}');
+
+ await backend.purge();
+
+ const store = await backend.createKeyValueStoreBackend({ name: 'hand-placed' });
+ expect(await readInput(store)).toBe('{"hand":"placed"}');
+ });
+
+ // An unnamed storage whose directory is named after its own id can only be reached through
+ // `{ id }` — not a run-scoped identifier, so it is not ours to empty.
+ test('leaves an unnamed storage directory named after its own id alone', async () => {
+ const ownDirectory = temporaryDirectory();
+ const firstRun = new FileSystemStorageBackend({ localDataDirectory: ownDirectory });
+ const dataset = await firstRun.createDatasetBackend({ name: 'seed' });
+ await dataset.pushData([{ from: 'another-tool' }]);
+ await firstRun.teardown();
+
+ // Turn the fixture into an unnamed storage living in an id-named directory.
+ const metadataPath = resolve(firstRun.datasetsDirectory, 'seed', '__metadata__.json');
+ const metadata = JSON.parse(await readFile(metadataPath, 'utf8')) as { id: string };
+ const { id } = metadata;
+ await writeFile(metadataPath, JSON.stringify({ ...metadata, name: null }));
+ await rename(resolve(firstRun.datasetsDirectory, 'seed'), resolve(firstRun.datasetsDirectory, id));
+
+ const secondRun = new FileSystemStorageBackend({ localDataDirectory: ownDirectory });
+ await secondRun.purge();
+
+ const reopened = await secondRun.createDatasetBackend({ id });
+ await expect(reopened.getData()).resolves.toMatchObject({ items: [{ from: 'another-tool' }] });
+ });
+});
+
+// Every crawler instance past the first gets an `__default___` queue from `openOwnedRequestQueue`,
+// so leaking those across runs makes a second crawler silently resume the previous run's requests even
+// with `purgeOnStart` enabled.
+describe('purgeDefaultStorages', () => {
+ afterEach(() => {
+ serviceLocator.reset();
+ });
+
+ test('clears the default queue', async () => {
+ serviceLocator.setStorageBackend(new FileSystemStorageBackend({ localDataDirectory: temporaryDirectory() }));
+
+ const defaultQueue = await RequestQueue.open();
+ await defaultQueue.addRequest({ url: 'https://example.com/stale' });
+ await purgeDefaultStorages();
+
+ expect(await defaultQueue.isEmpty()).toBe(true);
+ });
+
+ test('clears a crawler-owned alias queue left behind by a previous run', async () => {
+ const localDataDirectory = temporaryDirectory();
+ serviceLocator.setStorageBackend(new FileSystemStorageBackend({ localDataDirectory }));
+ const firstRun = await RequestQueue.open({ alias: '__default_1__' });
+ await firstRun.addRequest({ url: 'https://example.com/left-behind' });
+ await serviceLocator.getStorageBackend().teardown!();
+
+ serviceLocator.reset();
+ serviceLocator.setStorageBackend(new FileSystemStorageBackend({ localDataDirectory }));
+ await purgeDefaultStorages();
+
+ const secondRun = await RequestQueue.open({ alias: '__default_1__' });
+ expect(await secondRun.isEmpty()).toBe(true);
+ });
+});
diff --git a/test/core/storages/throttling_request_manager.test.ts b/test/core/storages/throttling_request_manager.test.ts
index 4b542ae2ca32..8493db573f2f 100644
--- a/test/core/storages/throttling_request_manager.test.ts
+++ b/test/core/storages/throttling_request_manager.test.ts
@@ -335,7 +335,7 @@ describe('ThrottlingRequestManager', () => {
expect(Date.now() - start).toBeLessThan(1000);
});
- test('picks up requests left in per-domain sub-queues by a previous run', async () => {
+ test('picks up requests left in per-domain sub-queues by a previous run (if purgeOnStart is not enabled)', async () => {
const domains = ['example.com'];
const firstRun = new ThrottlingRequestManager({ inner: await createQueue(), domains });