fix(deployer): decouple the restart backoff from the crash restart count - #300
Merged
Merged
Conversation
The crash restart backoff (2 * count * 1000 ms) was derived from crash_restart_count, a lifetime total that only ever grew, so the backoff grew without bound. After many crashes on a long-running system it could reach hours, effectively preventing the application from ever being restarted. Resetting crash_restart_count would fix the backoff but throw away the lifetime crash history reported in the UI and in the crash_restart notification. Instead, add a separate consecutive_crash_count that counts crashes since the application was last seen running. It drives the backoff and is reset in the :check_running handler, which only fires after the app has been up for timeout_app_ready (default 30s). crash_restart_count keeps its current meaning and is never reset. The backoff is also capped at 5 minutes as a safety net. Risk assessment: - Impact: an application that recovers and later crashes again restarts with the initial 2s backoff instead of an ever-growing one, and the backoff is bounded so a flapping app is always retried within 5 minutes. Reported crash counts are unchanged - Blast radius: deployer monitor only; the :check_running and :EXIT handlers plus one new field in the Monitor struct - Regression risk: low - the new field is internal to the monitor state, the counter reset only fires after the app is confirmed running, and the cap only affects extreme cases that previously produced multi-hour backoffs - Rollback: plain commit revert
thiagoesteves
force-pushed
the
thiagoesteves/fix-crash-restart-count-reset
branch
from
August 14, 2026 11:58
a92055d to
0f93cfc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
2 * count * 1000ms) was derived fromcrash_restart_count, a lifetime total that only ever grew, so the backoff grew without bound. After many crashes on a long-running system it could reach hours, effectively preventing the application from ever being restarted.crash_restart_countwould fix the backoff but throw away the lifetime crash history reported in the UI card and in thecrash_restartnotification. Instead, this adds a separateconsecutive_crash_countthat counts crashes since the application was last seen running. It drives the backoff and is reset in the:check_runninghandler, which only fires after the app has been up fortimeout_app_ready(default 30s).crash_restart_countkeeps its current meaning and is never reset.Reproduced by
consecutive_crash_count resets to 0 after application reports runninginapps/deployer/test/monitor_test.exs, which fails without the fix: the counter driving the backoff stayed at 1 after the app recovered, so the next crash waited 4s instead of 2s, and so on without bound.Test plan
mix test(767 tests, 0 failures)mix format --check-formattedmix credo --strict(no issues)mix dialyzer(0 errors -Deployer.Monitor.tgained a field)Risk assessment
:check_runningand:EXIThandlers, plus one new field in theDeployer.Monitorstruct.Deployer.Status, the notification payloads, and the web UI are untouched.timeout_app_ready, and the cap only affects extreme cases that previously produced multi-hour backoffs.🤖 Generated with Claude Code