Skip to content

fix(deployer): decouple the restart backoff from the crash restart count - #300

Merged
thiagoesteves merged 2 commits into
mainfrom
thiagoesteves/fix-crash-restart-count-reset
Aug 14, 2026
Merged

fix(deployer): decouple the restart backoff from the crash restart count#300
thiagoesteves merged 2 commits into
mainfrom
thiagoesteves/fix-crash-restart-count-reset

Conversation

@thiagoesteves

@thiagoesteves thiagoesteves commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • The crash restart backoff (2 * count * 1000 ms) was derived from crash_restart_count, a lifetime total that only ever grew, so the backoff grew without bound. After many crashes on a long-running system it could reach hours, effectively preventing the application from ever being restarted.
  • Resetting crash_restart_count would fix the backoff but throw away the lifetime crash history reported in the UI card and in the crash_restart notification. Instead, this adds a separate consecutive_crash_count that counts crashes since the application was last seen running. It drives the backoff and is reset in the :check_running handler, which only fires after the app has been up for timeout_app_ready (default 30s). crash_restart_count keeps its current meaning and is never reset.
  • The backoff is also capped at 5 minutes as a safety net.

Reproduced by consecutive_crash_count resets to 0 after application reports running in apps/deployer/test/monitor_test.exs, which fails without the fix: the counter driving the backoff stayed at 1 after the app recovered, so the next crash waited 4s instead of 2s, and so on without bound.

Test plan

  • mix test (767 tests, 0 failures)
  • mix format --check-formatted
  • mix credo --strict (no issues)
  • mix dialyzer (0 errors - Deployer.Monitor.t gained a field)

Risk assessment

  • Impact: An application that recovers and later crashes again restarts with the initial 2s backoff instead of an ever-growing one, and the backoff is bounded so a flapping app is always retried within 5 minutes. Reported crash counts are unchanged.
  • Blast radius: Deployer monitor only; the :check_running and :EXIT handlers, plus one new field in the Deployer.Monitor struct. Deployer.Status, the notification payloads, and the web UI are untouched.
  • Regression risk: Low - the new field is internal to the monitor state, the counter reset only fires after the app is confirmed running for timeout_app_ready, and the cap only affects extreme cases that previously produced multi-hour backoffs.
  • Rollback: Plain commit revert.

🤖 Generated with Claude Code

@thiagoesteves thiagoesteves self-assigned this Aug 14, 2026
The crash restart backoff (2 * count * 1000 ms) was derived from
crash_restart_count, a lifetime total that only ever grew, so the
backoff grew without bound. After many crashes on a long-running system
it could reach hours, effectively preventing the application from ever
being restarted.

Resetting crash_restart_count would fix the backoff but throw away the
lifetime crash history reported in the UI and in the crash_restart
notification. Instead, add a separate consecutive_crash_count that
counts crashes since the application was last seen running. It drives
the backoff and is reset in the :check_running handler, which only
fires after the app has been up for timeout_app_ready (default 30s).
crash_restart_count keeps its current meaning and is never reset. The
backoff is also capped at 5 minutes as a safety net.

Risk assessment:
- Impact: an application that recovers and later crashes again restarts
  with the initial 2s backoff instead of an ever-growing one, and the
  backoff is bounded so a flapping app is always retried within 5
  minutes. Reported crash counts are unchanged
- Blast radius: deployer monitor only; the :check_running and :EXIT
  handlers plus one new field in the Monitor struct
- Regression risk: low - the new field is internal to the monitor
  state, the counter reset only fires after the app is confirmed
  running, and the cap only affects extreme cases that previously
  produced multi-hour backoffs
- Rollback: plain commit revert
@thiagoesteves
thiagoesteves force-pushed the thiagoesteves/fix-crash-restart-count-reset branch from a92055d to 0f93cfc Compare August 14, 2026 11:58
@thiagoesteves thiagoesteves changed the title fix(deployer): reset crash_restart_count and cap backoff fix(deployer): decouple the restart backoff from the crash restart count Aug 14, 2026
@thiagoesteves thiagoesteves moved this to In Progress in DeployEx Project Aug 14, 2026
@thiagoesteves
thiagoesteves merged commit 03c7fea into main Aug 14, 2026
1 check passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in DeployEx Project Aug 14, 2026
@thiagoesteves
thiagoesteves deleted the thiagoesteves/fix-crash-restart-count-reset branch August 14, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant