Skip to content

feat(risk-guard): tail-risk guard P1 — SPX speed / account ratio / BLIND alerts (no orders) - #268

Merged
moremeds merged 18 commits into
masterfrom
feat/tail-risk-guard
Nov 8, 2026
Merged

feat(risk-guard): tail-risk guard P1 — SPX speed / account ratio / BLIND alerts (no orders)#268
moremeds merged 18 commits into
masterfrom
feat/tail-risk-guard

Conversation

@moremeds

@moremeds moremeds commented Nov 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

P1 of the tail-risk guard (spec docs/superpowers/specs/2026-11-08-tail-risk-guard-design.md, plan docs/superpowers/plans/2026-11-08-tail-risk-guard-p1.md). Alerts only: it never places, cancels or modifies an order (test_risk_guard_no_order_path.py). Off by default (XENON_RISK_GUARD=1).

  • Guard loop (src/xenon/api/services/risk_guard/): every 1 s it reads SPX + SPY streaming tickers on the IB data role and every 30 s the latest account snapshot.
    • Market tiers: WATCH / FAST / CRASH from 15 / 30 / 60 m speed, day return, key levels, short-strike distance, long leg ITM and cushion.
    • BLIND causes: stale or divergent feed, orders halted, data role down, upstream lost, DB down or slow, stale or unreadable snapshot.
    • Ratio alarms: leverage and flow-adjusted drawdown, plus a 09:00 ET daily line.
    • Sends ntfy + Resend first, then writes xenon.risk_events (no write under XENON_READ_ONLY).
  • API: GET /risk/state, POST /risk/incidents/{key}/ack (internal token, top-level reason_code), a /health risk_guard block, and runtime settings XENON_RISK_GUARD / XENON_RISK_THRESHOLDS / XENON_RISK_KEY_LEVELS. Migration 2026_11_09_risk_events. Shutdown budget is now 58 s < 60 s.
  • UI: RiskGuardBanner with an ACK button and a details panel. Screenshots: output/playwright/risk-guard-2026-11-08*.png (local).
  • Calendar: NYSE early closes and us_equity_session (rth / extended / overnight / closed).
  • Relay: a stale SPX / NDX / RUT / DJX / VIX snapshot in RTH gets last: null, and the stale-restart loop backs off.
  • Replay (docs/research/2026-11-08-risk-guard-replay.md, UW SPY bars):
    • On 2026-11-05 the tiers fired at WATCH 09:42, FAST 10:11 and CRASH 10:46 ET. All three came from the day return; the speed windows alone peaked at 30 m −1.43 % and 60 m −2.38 %.
    • Over 12 months: WATCH 50 days, FAST 7, CRASH 1.
  • Docs: runbook docs/runbooks/risk-guard.md, src/xenon/api/CLAUDE.md, CHANGELOG.

Review

A different-model (Sonnet) review found 9 items.

  • Fixed: data role never reconnected; DB reads unbounded; false BLIND at 09:30; speed triggers silent after a feed gap; error isolation; relay mask too wide; then a shared DB budget found on re-check.
  • Documented as limits: Ticker.time is a receive time; the relay has no holiday table, so the masked indexes show --- on holidays.
  • Left open (low): relay snapshot path, restart-streak reset, and the sync runtime_settings read on the event loop (a project-wide pattern).

Test plan

  • uv run python scripts/infra/dev/run_pytest_affected.py --base origin/master → 973 passed (local scratch PG)
  • Risk guard tests: 132 passed
  • npm test --prefix scripts/infra/ib_realtime → 124 passed
  • Vitest 2005 passed. The 8 order-e2e.test.ts failures also fail on origin/master locally (FastAPI harness env).
  • web lint: 0 errors. Typecheck: only stale .next cache errors locally.
  • Playwright e2e/risk-guard-banner.spec.ts passed, with screenshots
  • Six order-path guards OK
  • CI green
  • Real ntfy / Resend delivery (needs the ntfy topic and Resend token)
  • Dead-man checker host and kill test
  • SPX index live entitlement check on the live account (RTH)

Update 2026-11-09

  • Extra indexes (004ea0f9): runtime setting XENON_RISK_WATCH_SYMBOLS over NDX / RUT / DJX (IB exchanges NASDAQ / RUSSELL / CBOE verified with reqContractDetails on the live gateway). Own market:<SYM> incident and ack, same speed thresholds, capped at FAST, no BLIND. SPX + SPY fixed; VIX deferred (rise rule open). Design: Fable advice.
  • Astra review (gpt-6-astra, 3 passes; pass 1 CHANGES NEEDED with 6 majors, pass 2 CHANGES NEEDED with 3, pass 3 APPROVE): ack applies only to the incident, tier, reference inputs and send revision it was written for (else 409 RISK_INCIDENT_CHANGED); a send that reached no channel retries every 30 s (max 10) with a per-key revision; IB 1101 re-subscribes; session high/low per trading date; BLIND BASIS_UNAVAILABLE; risk_events writes 5 s deadline and max 4 in flight; /risk/state events read 5 s deadline. Each fix has a regression test; the ack-race, next-day-session and basis tests were confirmed red on 004ea0f9.

🤖 Generated with Claude Code

chenxi added 18 commits November 8, 2026 22:38
chenxi
docs: tail-risk guard spec v3, P1 plan, P0 findings
9e26a85
chenxi
chore(checks): skip gitignored tmp/ in three repo-walking guards
e4c245c
chenxi
feat(calendar): NYSE early closes and US equity session (risk guard P…
3127465 
…1 task 3)
chenxi
feat(risk-guard): windows, tiers/incidents, account ratios, notifier …
81aa406 
…(P1 tasks 1,4,5,6)
chenxi
fix(relay): withhold stale cash-index last in RTH, back off stale rec…
78d7360 
…onnects, log IB info codes

P0 finding (docs/research/2026-11-08-risk-guard-p0.md §1): on 11-03..05 IB sent no SPX ticks in RTH and the relay published the prior close as last with HTTP 200; the stale-data check reconnected every ~2.5 min (166/day) because the tick clock was not reset on connect.
chenxi
fix(risk-guard): max loss from legs, re-rise and worsening seeding, n…
cadced7 
…otifier hardening (review)

Same-expiry spreads arrive from ib_sync as one 'Combo (N legs)' row without max_risk, so leverage under-counted the user's 11-27 spreads. Max loss is now the worst expiry P&L over {0} + strikes; unbounded or unreadable positions are counted, not summed.
chenxi
feat(risk-guard): risk_events table, queries, runtime settings (P1 ta…
e8a6dd1 
…sks 7, 8)
chenxi
feat(risk-guard): IB SPX/SPY sources, freshness and feed health (P1 t…
602f6a8 
…ask 2)
chenxi
feat(risk-guard): service loop, heartbeat, lifespan wiring, /risk rou…
f33e907 
…tes, /health block (P1 tasks 9, 10)
chenxi
feat(risk-guard): risk banner with ack and details panel (P1 task 11)
4840b51
chenxi
feat(risk-guard): replay harness on UW SPY bars + report (P1 task 12)
75314bc
chenxi
docs(risk-guard): runbook, api CLAUDE.md, CHANGELOG, compose env note…
dcba197 
…, spec status (P1 task 13)
chenxi
fix(risk-guard): review fixes — data-role retry, bounded DB reads, op…
1e29e1e 
…en grace, warming sources, error isolation, narrower relay mask
chenxi
fix(risk-guard): separate DB budget for the drawdown basis (review re…
92c18b7 
…-check)
chenxi
feat(risk-guard): configurable extra indexes NDX/RUT/DJX with own FAS…
004ea0f 
…T-capped incidents

XENON_RISK_WATCH_SYMBOLS (/admin) adds single-source cash indexes (IB exchanges
verified with reqContractDetails 2026-11-08). Each gets market:<SYM> with its
own ack, same speed thresholds, capped at FAST, no BLIND when stale. One new
subscription per read; removed symbols cancel their line. SPX/SPY stay fixed;
VIX deferred (rise rule open). Design advice: Fable.
chenxi
fix(risk-guard): Astra review — ack identity check, redelivery, 1101 …
3b8cd49 
…resubscribe, per-day session, basis BLIND, bounded event writes
chenxi
style(risk-guard): ruff format test
ee254fc
chenxi
fix(risk-guard): Astra re-review — ack keeps its pre-write reference …
7e9a413 
…and send revision, retry only drops on a later ack, per-key dispatch revision
@moremeds
moremeds merged commit 36ed13f into master Nov 8, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@moremeds