Skip to content

fix: compute pcr_oi from open interest, not UW's volume ratio - #570

Merged
moremeds merged 5 commits into
mainfrom
fix/pcr-oi-open-interest
Nov 8, 2026
Merged

moremeds merged 5 commits into
mainfrom
fix/pcr-oi-open-interest

Conversation

@moremeds

@moremeds moremeds commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Problem

MarketAggregates.pcr_oi was set from UW screener put_call_ratio. UW's spec (docs/uw-samples/unusual_whales_api_spec.yaml, /api/screener/stocks) defines that field as put volume / call volume. So pcr_oi has been a copy of pcr_vol:

  • prod pcr_history: 38,376 rows (2026-01-02 → 2026-11-06); 38,342 have pcr_oi == pcr_vol, 0 differ, 34 have NULL pcr_vol.
  • Affected readers: watchlist "PCR (OI)" tile, pcr_delta_30d, market_overview.positioning, the Trade Insights AI prompt input.
  • Example: TSLA showed pcr_oi 0.9397 (= pcr_vol); put_oi/call_oi is ~0.80.

Fix

  • pipeline.aggregates_from_screener() computes pcr_oi = put_oi / call_oi and pcr_vol = put_vol / call_vol from the screener's own counts. Zero puts → 0; missing/zero calls → NULL. run_single_stock uses it.
  • Repository.repair_pcr_history_oi() (in _MarketDataMixin) + scripts/backfill/pcr_history_oi_repair.py rewrite pre-fix rows from options_volume_daily. The match is the latest session <= snapshot_date within --max-lag-days (default 4), because the UTC-stamped snapshot_date can be a day ahead of the ET session (weekend rows). No match → NULL, never the old volume ratio. Dry run by default; idempotent; bounded by --before.
  • A script, not a migration: migration DML must be insert-only.

Post-deploy (manual, needs operator confirmation)

Pass the day after the deploy as --before, and run only after the deploy day's 18:15 ET flow refresh has written options_volume_daily for that day.

python scripts/backfill/pcr_history_oi_repair.py --before <day after deploy>          # dry run
python scripts/backfill/pcr_history_oi_repair.py --before <day after deploy> --apply

Prod check (read-only, 2026-11-08): of 17,203 rows that map to their last-writing scan run, the repair reproduces that scan's own OI ratio on 16,661; 275 match an earlier session (morning-only last scans saw pre-publish OI); 267 have no exact match (248 from notes-tagged, non-full-scan runs).

Until it runs, pcr_delta_30d mixes the old volume ratio with the new OI ratio.

Tests

  • tests/unit/test_pipeline_aggregates_from_screener.py — UW's AAPL spec example (OI ratio ≠ volume ratio), zero/missing counts.
  • tests/integration/storage/test_pcr_history_oi_repair.py — dry run rolls back, apply values, weekend row takes Friday, no-source → NULL, post-cutover row untouched, rerun changes 0.
  • Local: full lint+unit gate green (3195 unit passed); integration 5 passed; dry run on option_wizard_local: 34,657 in scope, 34,652 with OI source, 5 → NULL.

Review

Tribunal: Opus 5.5 lead + Fable 5.1 + GPT-6-Astra (herdr panes). Fixed: cutoff guidance (both), negative --max-lag-days now raises (Astra), stamping docstring (Fable, Astra), tests for zero call OI + per-ticker isolation (Fable; mutation-checked). Astra's "repair picks a later session" CRITICAL was withdrawn after the prod evidence above. All findings verified resolved by the raising seat.

Not in scope

snapshot_date uses date.today() in a UTC container, so an ad hoc rescan after ~20:00 ET stamps the next day. scan_runs.aggregates JSON history keeps the old pcr_oi value; nothing reads old runs for PCR history. Separate issues.

🤖 Generated with Claude Code

chenxi added 4 commits November 8, 2026 10:24
The scan stored UW's screener put_call_ratio as MarketAggregates.pcr_oi,
but UW's spec defines that field as put volume / call volume. pcr_oi has
been a copy of pcr_vol in every pcr_history row since 2026-01-02, and
pcr_delta_30d, the watchlist "PCR (OI)" tile and the Trade Insights AI
prompt all read the volume ratio under an OI label.

- pipeline.aggregates_from_screener derives both ratios from the
  screener's own OI and volume counts (zero puts is 0, zero calls is no
  ratio).
- Repository.repair_pcr_history_oi + scripts/backfill/pcr_history_oi_repair.py
  rewrite pre-fix rows from options_volume_daily (latest session within
  --max-lag-days of snapshot_date, else NULL). Dry run by default,
  idempotent, bounded by --before.
- Cutoff guidance: pass the day after the deploy as --before; the deploy
  day can still hold old-code rows, and the repair recomputes the same
  value for rows the fixed scan wrote.
- Reject a negative --max-lag-days, which would NULL every in-scope row.
- Docstring: scheduled scans stamp the ET session date; only ad hoc
  evening or weekend rescans land on a non-session date.
- Tests: zero call OI sets NULL, a second ticker's session does not leak
  through the correlated subquery, negative lag is rejected.
@moremeds
moremeds merged commit 75d5431 into main Nov 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant