Add a hydration timeout to avoid blocking enqueue for too long

[udnay]

We just shipped eager startup hydration: JobStoreShard::open now awaits ConcurrencyCounts::hydrate_all, which scans every concurrency holder in durable storage and populates the in-memory cache before the shard goes live. This guarantees correct grant decisions from the first operation but pays the full scan cost synchronously.

For shards with very large holder sets that scan can take a long time. We want to soft-cap the hydration window so the shard becomes responsive quickly without sacrificing correctness:

1. Hydration runs in a background task with a configurable timer T.
2. After T fires (or hydration completes, whichever comes first), the shard starts accepting enqueues.
3. Until hydration is fully done, the shard defers ticket granting: every concurrency-limited enqueue creates a durable TicketRequest instead of a holder. The grant scanner stays idle.
4. When hydration finishes, the grant scanner wakes up and drains the backlog in priority order.
5. Default behavior is unchanged: with no timer configured, open blocks on hydration exactly like it does today.

This preserves the safety property the Alloy spec already documents (omittedQueuesAreSafe) and adds a new operational property: "no holder is created while grants_enabled is false."

---

Note

High Risk
Changes shard open, concurrency granting, and enqueue/import gating on the critical path; incorrect timing could overcommit queues or reject traffic until hydration completes.

Overview
Adds optional startup_hydration_timeout_ms so shards with eager startup hydration (hydrate_all_at_startup) no longer have to block open on a full holder scan. Unset (default): behavior matches today—open waits for hydration, then enqueues and grants are allowed. Set: hydration runs in the background with a timer; after the timeout (or when hydration finishes, whichever applies for enqueues), the shard accepts work while grants_enabled stays false until hydration succeeds.

Two startup gates: accepting_enqueues blocks enqueue / import_jobs with ShardHydrating (gRPC unavailable); grants_enabled makes try_reserve_internal fail closed so concurrency-limited work becomes durable TicketRequests instead of holders. The grant scanner waits on that flag, then reconciles and drains; wake_grant_scanner runs when grants flip on. Failed hydration retries with grants still off; open failures call stop_background_tasks so timers/tasks don’t leak.

Config is threaded through factory, settings, benches, and test helpers; specs/job_shard.als documents the grant-gate invariant. New integration tests cover gate ordering, backlog drain, and pre-gate rejections.

^{Reviewed by Cursor Bugbot for commit f5d5e04. Bugbot is set up for automated code reviews on this repo. Configure here.}

[udnay]

This change is part of the following stack:

• Hydrate all queues at startup #308
  • Add a hydration timeout to avoid blocking enqueue for too long #309 ◀

_{Change managed by git-spice.}

[kirinrastogi]

I think the typescript client should be updated to retry on this error if we do go this route

[kirinrastogi]

I don't think we should track this, the shard factory will return grpc Unavailable until this open_with_resolved_store completes. We should continue to use that signal. It is redundant to mark the shard as available but then return a new ShardHydrating error. And it complicates the routing clients.

We should measure the hydration time with your other PR, and if it is reasonable then we should do it synchronously on shard open, or let it run for ~5s or so before marking the shard as open. Some downtime here is acceptable because clients will retry their enqueues/leasetasks/report outcome rpcs

because of the dashmap usage hydration should work with enqueues at the same time

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9b4c375. Configure here.}