Skip to content

Add PromoteMaster command for manual master promotion - #2246

Open
alytaphoenix wants to merge 1 commit into
exo-explore:mainfrom
alytaphoenix:feature/promote-master
Open

Add PromoteMaster command for manual master promotion#2246
alytaphoenix wants to merge 1 commit into
exo-explore:mainfrom
alytaphoenix:feature/promote-master

Conversation

@alytaphoenix

Copy link
Copy Markdown

Summary

Adds an API endpoint to force a specific node to win the next master election, for operators who want to move mastership without waiting for organic re-election (e.g. moving master off a node they're about to take down for maintenance).

  • POST /master/promote/{node_id} sends a PromoteMaster command, rejected with 409 if any instances are running (every node's worker/download-coordinator restarts on master change — see exo.main._elect_loop — so this is idle-only) and 404 if the node isn't in the topology.
  • Election._force_promote boosts the target node's seniority strictly above the highest seniority it has ever observed from any peer (floored at the same FORCE_MASTER_SENIORITY that --force-master uses at startup), then triggers a new election round.

A fixed constant here would let a second promote tie the first on seniority and fall through to the commands_seen tiebreak, which favours whichever node has been master longest — silently no-opping the newer promotion. Tracking the highest seniority observed and always going one higher keeps repeated promotions, and promotions away from a --force-master node, working. This was caught with a regression test (test_second_promote_still_wins_over_previously_promoted_master) that reproduces the exact scenario; verified by reverting to the fixed-constant version and confirming the test fails as predicted, then passes with the fix.

Known limitations

  • Sticky master: like --force-master, a promoted node's seniority only ever grows, so it stays master for the rest of its process lifetime and won't hand back automatically. A later PromoteMaster targeting a different node still works correctly (see above). If the promoted node crashes, peers re-elect normally, and a restarted node rejoins at seniority 0.
  • Not yet exercised on a real multi-node cluster — covered by unit tests against Election and the API handler in isolation.
  • No dashboard UI for this yet (API-only); a "promote to master" control in TopologyGraph is a natural follow-up.

Test plan

  • uv run basedpyright — 0 new errors (baseline unchanged: 319 errors, 110 warnings)
  • uv run ruff check — passes
  • uv run ruff format — applied
  • uv run pytest src/exo/shared/tests/test_election.py src/exo/api/tests/test_promote_master.py — 23 passed, including a regression test that fails against the pre-fix code
  • Broader uv run pytest src (excluding pre-existing environment gaps unrelated to this change — mlx not installed in this dev environment) — 330 passed, 3 skipped
  • Manual verification on a real multi-node cluster

🤖 Generated with Claude Code

https://claude.ai/code/session_011rjSfwDBTkmySmfU6NgHKF

Adds an API endpoint to force a specific node to win the next master
election, for operators who want to move mastership without waiting
for organic re-election (e.g. moving master off a node they're about
to take down for maintenance).

- POST /master/promote/{node_id} sends a PromoteMaster command,
  rejected with 409 if any instances are running (every node's worker
  restarts on master change, so this is idle-only) and 404 if the
  node isn't in the topology.
- Election._force_promote boosts the target node's seniority strictly
  above the highest seniority it has ever observed from any peer
  (floored at the same FORCE_MASTER_SENIORITY --force-master uses at
  startup), then triggers a new round. A fixed constant here would let
  a second promote tie the first on seniority and fall through to the
  commands_seen tiebreak, which favours whichever node has been master
  longest -- silently no-opping the newer promotion. Tracking the
  observed max and always going one higher keeps repeated promotions,
  and promotions away from a --force-master node, working. Verified by
  reverting to the fixed-constant version and confirming the new
  regression test fails exactly as predicted, then passes with the fix.

Known limitations:
- Sticky master: like --force-master, a promoted node's seniority only
  ever grows, so it stays master for the rest of its process lifetime
  and won't hand back automatically. A later PromoteMaster targeting a
  different node still works (see above). If the promoted node
  crashes, peers re-elect normally, and a restarted node rejoins at
  seniority 0.
- Not yet exercised on a real multi-node cluster -- covered by unit
  tests against Election and the API handler in isolation.
- No dashboard UI for this yet (API-only); a "promote to master"
  control in TopologyGraph is a natural follow-up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011rjSfwDBTkmySmfU6NgHKF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant