Skip to content

feat(risk-guard): acks survive a guard restart (P2.0) + P2 plan - #272

Open
moremeds wants to merge 7 commits into
masterfrom
feat/risk-guard-ack-persist
Open

moremeds wants to merge 7 commits into
masterfrom
feat/risk-guard-ack-persist

Conversation

@moremeds

@moremeds moremeds commented Nov 9, 2026

Copy link
Copy Markdown
Owner

Summary

Tail-risk guard P2.0: an acked incident no longer re-pages after a guard restart (deploy, api recreate). Before, every deploy re-paged the acked FAST incident every 30 s until someone acked it again (seen twice on 2026-11-09).

  • The guard stores the set of applied acks (incident id, sent and highest severity, worsening reference) as ack_state rows in xenon.risk_events. No schema change.
  • One writer per process appends the whole set whenever it differs from the last committed one: at start, at once when an ack is applied, and when an acked incident rises, clears or has a worsening delivered. Rows carry a process nonce plus a sequence number; restore takes the newest process's highest sequence, so a late write never wins.
  • At shutdown flush_state writes the final set first (0.6 s, outside the 4-write cap).
  • Before its first evaluation with a resolved account scope, the guard restores the latest set of the current ET date. A rise or a worsening during the downtime alerts on the first tick (no restored cooldown).
  • A worsening reference is stored only after its alert reached a channel; an ack drops pending worsening references.
  • An account change clears the acks and restores the new account's set. An ack begun under another account is refused (changed), and the writer never publishes an account before its restore.
  • Every restored field is type-checked; an invalid entry is dropped and logged (that incident pages again). A failed read restores nothing (fail loud).
  • GET /risk/state hides ack_state rows.

Also includes the P2 plan, docs/superpowers/plans/2026-11-09-tail-risk-guard-p2a.md (v4): IB and Futu atomic closes, delivered as P2.1–P2.5. That plan is not implemented here.

Limits (runbook)

  • Incidents that were not acked still re-open and page after a restart.
  • Leverage / drawdown ratio alarms and the daily line send once more after a restart.
  • A set whose writes all failed before the process died is lost: the older set is restored, and a higher tier still alerts at once.

Review

  • Astra (gpt-6-astra) plan review: four rounds, final SHIP.
  • Astra code review: five rounds, final SHIP. Every round's findings were fixed, each with a regression test.
  • Mutation red checks: each fix was reverted in turn, and its regression test failed every time.

Test plan

  • uv run pytest -p no:xdist src/xenon/api/tests/test_risk_guard_*.py scripts/tests/test_risk_guard_queries.py src/xenon/api/tests/test_read_only_mode.py src/xenon/api/tests/test_health_observability.py scripts/tests/test_runtime_settings.py → 356 passed (local scratch DB)
  • Order-path guards, ruff check --select F src scripts, ruff format
  • CI
  • After deploy: ack the live incident, recreate the api, and confirm it does not re-page

The guard places, cancels and modifies no order (test_risk_guard_no_order_path.py).

🤖 Generated with Claude Code

chenxi added 7 commits November 9, 2026 01:39
Restore incidents acked earlier on the same ET date from risk_events before
the first evaluation; record close and ack_void lifecycle rows (retried);
per-process incident id prefix.
…a review)

Replace folding of notification rows (write order != lifecycle order,
unapplied acks, lost close rows) with a full acked-state snapshot written by
one serialized writer on every change; validate every restored field; keep
the latest worsening reference; uuid incident-id nonce.
…refs only (Astra review 2)

Write the acked set at once on ack and directly at shutdown (no write cap);
proc nonce + seq so a late write never wins; restore waits for a resolved
scope and a scope change clears acks; persist a worsening reference only
after its alert reached a channel; no restore cooldown; require complete,
float-safe references.
…nding worsening refs; restore flag after the read (Astra review 3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant