[moremeds]

What this adds

Before v0.13.28 (#546) the nightly capture kept at most 500 prints per ticker, day and source. The new dark_lit_backfill job re-pages the cut history through the same capture_dark_lit_for, so there is still one writer.

• Scope: every active-watchlist ticker (removed_at IS NULL) × trading session, from UW's earliest available date through LAST_CUT_DATE = 2026-10-05. Oldest first, because UW's 730-trading-day window expires the oldest days first.
• Skipped: a ticker-day already finished (dark_lit_backfill_progress), and one already complete (both sources under 500 prints, so never cut). Cut (≥500 in either source) or missing ticker-days are fetched.
• Calendar: SPY daily bars from apex (one call per run, zero UW). The warm-store spine (daily_ohlc SPY) only starts 2025-04-14 on prod, so it cannot cover 2023.
• Schedule: daily 22:30 ET, pinned to uw-0, UW_SCAN_DARK_LIT_BACKFILL_ENABLED default false. No behaviour change on deploy.

Budget

• Caps per UW budget day (UTC day, the account counter's 00:00 UTC = 20:00 ET reset):

  Setting	Default
  UW_SCAN_DARK_LIT_BACKFILL_WEEKDAY_MAX_CALLS	15,000
  UW_SCAN_DARK_LIT_BACKFILL_SATURDAY_MAX_CALLS	60,000
  UW_SCAN_DARK_LIT_BACKFILL_SUNDAY_MAX_CALLS	60,000

  The Friday and Saturday 22:30 ET runs fall in UTC Saturday and Sunday; the Sunday-evening run is UTC Monday (weekday cap).

• Spend = this job's own rows in external_api_requests for the budget day of the run's start. Every attempt is a row, so retries count. A run that crosses UTC midnight keeps counting instead of resetting, which can only stop it early.

• Never exceeds the cap: it stops when a worst-case ticker-day (2 sources × 60-page cap = 120 calls) would not fit, leaving at most 120 calls unused.

• Governor: research_budget_ok (research ceiling + total guard) is checked every 25 ticker-days; a "no" stops the run. Live jobs keep priority.

• Effective weekend spend is capped at ~45k/day by the current research ceiling. Prod has UW_RESEARCH_DAILY_CEILING=45000, below the 60k weekend cap. Raising it is the user's decision; this PR does not change prod env.

• Other consumer on the key: the account counter reached 57,004 (UTC Sat 2026-09-26) and 91,629 (UTC Sat 10-03) while argon logged only 1,019 / 1,490 calls. UW_TOTAL_DAILY_GUARD=105000 halts every pool, so weekend headroom can be far below 45k.

Resume, window, run log

• Resume: progress is stored per ticker-day (done / unavailable); a restart skips finished rows.

• Out-of-window: UW answers 403 historic_data_access_missing with the earliest available date in the message (probed 2026-10-05: earliest 2023-11-02). The job marks that ticker-day unavailable, skips every ticker on that date and everything before the named earliest date, and future runs skip dates at or before the newest unavailable date.

• dark_lit_backfill_runs: one row per run with the budget day, quota, calls, ticker-days done, unavailable, errors, remaining pending ticker-days, and stop_reason.

  stop_reason	Meaning
  quota	the day's cap is reached
  research_budget	the governor said no
  error_streak	20 consecutive failures, so an outage cannot burn the quota
  exhausted	nothing left to fetch
  calendar_unavailable	apex is down

• Migration 161: creates the two bookkeeping tables only, with no data change. Both are registered as provenance in the data-gap registry, and the policy doc is regenerated.

Horizon (estimate, not a promise)

• Size: 171 tickers × ~735 sessions ≈ 125k ticker-days. The new table starts 2026-01-02, so nearly everything before is missing.
• Calls: at ~10 calls per ticker-day, ≈ 1.2M calls.
• Throughput: at 15k × 5 + ~45k × 2 ≈ 165k/week, about 7–8 weeks.
• Window race: oldest-first clears ~100+ sessions/week against 5 expiring, so it outruns the window.

LAST_CUT_DATE

• Value: 2026-10-05, the day v0.13.28 (the paged capture) was cut. Including it costs at most one re-page of a complete day; excluding a cut day would lose it.
• When to change it: raise it only if the deploy landed after the 2026-10-06 18:55 ET capture.

Tests

• Unit tests/unit/worker/test_dark_lit_backfill.py: quota by UTC weekday (Friday-evening ET → Saturday cap); pending list oldest first; complete vs cut rule; finished and out-of-window skips; parsing UW's verbatim 403 message.
• Integration tests/integration/worker/test_dark_lit_backfill.py (real migrated DB, stub UW):
  • Run 1 stops on quota after the out-of-window probe plus 3 ticker-days. The out-of-window date is probed once and then skipped for the other ticker.
  • Run 2 resumes, fetching only the remaining date, and ends exhausted.
  • Progress, run rows and print counts are checked.
  • A governor "no" stops before any call.
  • spent_today counts only this job's UW rows in the run's UTC budget day (not another job's, not massive, not the previous day). Spend already recorded there stops a run on quota before any call. Mutation: spent_today returning 0 fails both.
  • Mutation check: removing the progress write, or the out-of-window skip, each fails the test.
• Goldens regenerated on purpose: scheduler (new job on all_enabled/uw-0 and all_enabled/all-0 only), settings (+4 fields), lock keys (+dark_lit_backfill 10806).
• Results: unit 3,169 passed. Integration (migrations, DML replay, data-gap full coverage, uw-alpha capture, this job): 32 passed. Ruff clean.
• Prod (read-only): the candidate query (GROUP BY market_date, ticker over the 16.8M-row table) is an Index Only Scan, 0.83 s.

🤖 Generated with Claude Code