Add PromoteMaster command for manual master promotion - #2246
Open
alytaphoenix wants to merge 1 commit into
Open
Conversation
Adds an API endpoint to force a specific node to win the next master
election, for operators who want to move mastership without waiting
for organic re-election (e.g. moving master off a node they're about
to take down for maintenance).
- POST /master/promote/{node_id} sends a PromoteMaster command,
rejected with 409 if any instances are running (every node's worker
restarts on master change, so this is idle-only) and 404 if the
node isn't in the topology.
- Election._force_promote boosts the target node's seniority strictly
above the highest seniority it has ever observed from any peer
(floored at the same FORCE_MASTER_SENIORITY --force-master uses at
startup), then triggers a new round. A fixed constant here would let
a second promote tie the first on seniority and fall through to the
commands_seen tiebreak, which favours whichever node has been master
longest -- silently no-opping the newer promotion. Tracking the
observed max and always going one higher keeps repeated promotions,
and promotions away from a --force-master node, working. Verified by
reverting to the fixed-constant version and confirming the new
regression test fails exactly as predicted, then passes with the fix.
Known limitations:
- Sticky master: like --force-master, a promoted node's seniority only
ever grows, so it stays master for the rest of its process lifetime
and won't hand back automatically. A later PromoteMaster targeting a
different node still works (see above). If the promoted node
crashes, peers re-elect normally, and a restarted node rejoins at
seniority 0.
- Not yet exercised on a real multi-node cluster -- covered by unit
tests against Election and the API handler in isolation.
- No dashboard UI for this yet (API-only); a "promote to master"
control in TopologyGraph is a natural follow-up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011rjSfwDBTkmySmfU6NgHKF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an API endpoint to force a specific node to win the next master election, for operators who want to move mastership without waiting for organic re-election (e.g. moving master off a node they're about to take down for maintenance).
POST /master/promote/{node_id}sends aPromoteMastercommand, rejected with409if any instances are running (every node's worker/download-coordinator restarts on master change — seeexo.main._elect_loop— so this is idle-only) and404if the node isn't in the topology.Election._force_promoteboosts the target node's seniority strictly above the highest seniority it has ever observed from any peer (floored at the sameFORCE_MASTER_SENIORITYthat--force-masteruses at startup), then triggers a new election round.A fixed constant here would let a second promote tie the first on seniority and fall through to the
commands_seentiebreak, which favours whichever node has been master longest — silently no-opping the newer promotion. Tracking the highest seniority observed and always going one higher keeps repeated promotions, and promotions away from a--force-masternode, working. This was caught with a regression test (test_second_promote_still_wins_over_previously_promoted_master) that reproduces the exact scenario; verified by reverting to the fixed-constant version and confirming the test fails as predicted, then passes with the fix.Known limitations
--force-master, a promoted node's seniority only ever grows, so it stays master for the rest of its process lifetime and won't hand back automatically. A laterPromoteMastertargeting a different node still works correctly (see above). If the promoted node crashes, peers re-elect normally, and a restarted node rejoins at seniority 0.Electionand the API handler in isolation.TopologyGraphis a natural follow-up.Test plan
uv run basedpyright— 0 new errors (baseline unchanged: 319 errors, 110 warnings)uv run ruff check— passesuv run ruff format— applieduv run pytest src/exo/shared/tests/test_election.py src/exo/api/tests/test_promote_master.py— 23 passed, including a regression test that fails against the pre-fix codeuv run pytest src(excluding pre-existing environment gaps unrelated to this change —mlxnot installed in this dev environment) — 330 passed, 3 skipped🤖 Generated with Claude Code
https://claude.ai/code/session_011rjSfwDBTkmySmfU6NgHKF