Skip to content

ThrottlingRequestManager never reopens its sub-queues, so a restart strands throttled requests #2142

Description

@janbuchar

_get_or_create_sub_manager is only reached from add_request and add_requests, but every read
path — fetch_next_request, is_empty, is_finished, get_handled_count, get_total_count,
purge, drop — iterates _sub_managers. On a fresh process that dict is empty.

  • Requests a previous run left in a throttled-<domain> queue stay invisible until something adds a
    new request for that same domain. Until then fetch_next_request takes its
    if not self._sub_managers: return None short-circuit and the crawl reports itself finished with
    work still queued.
  • purge() and drop() skip them for the same reason, so they cannot be cleared either.

The sub-queues live under a stable alias specifically so they outlive the process, which this
defeats. The TS port opens every configured domain up front rather than on insert
(apify/crawlee#3741).

From #1762.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions