Raft Cluster: Automatic learner promotion (#4392)

[quanyeyang]

Closes #4392.

Follow-up to #4094, which introduced non-voting Raft members (learners).

This PR adds automatic promotion of caught-up learners and, as part of that
work, serializes quorum-changing membership transitions so that Cluster V2
has at most one distinct voting-membership change in flight at a time.

Motivation

After #4094, newly joined nodes enter the cluster as non-voting learners and
must be promoted manually with CLUSTER ADDVOTER.

For the common bootstrap case, this requires the administrator to manage the
voter set explicitly even when no special voter placement policy is needed.

This PR makes the default behavior self-managing:

• the Raft leader promotes eligible learners automatically;
• the voting set converges toward a target of 5 voters;
• promotion reuses the existing ADD_VOTER Raft entry and apply path.

No new node flag or membership entry type is introduced.

Automatic learner promotion

The leader runs a one-way reconciliation controller:

voters < target
        |
        v
no quorum-changing membership
transition in flight
        |
        v
select one caught-up learner
        |
        v
propose ADD_VOTER

The initial target is hard-coded to:

RAFT_TARGET_VOTERS = 5

The controller is intentionally one-way:

• only the current leader makes promotion decisions;
• it never automatically demotes a voter;
• once size >= RAFT_TARGET_VOTERS, the controller is a no-op;
• failed voters still count toward the configured voter set and are not
  automatically replaced.

A learner is eligible for automatic promotion only when it:

• has completed NODE_JOIN and is no longer a MEET node;
• is not marked failed;
• has an active cluster link;
• has caught up to the leader's current log
  (match_index >= raftLogLastIndex()).

The actual membership mutation still goes through the existing
ADD_VOTER apply path.

Serialized quorum membership changes

Automatic promotion also exposes a broader issue with overlapping membership
changes in the current config-on-apply model.

server.cluster->size changes only when a membership entry is applied.
Without serialization, multiple membership changes could be proposed while
all of them still observe the same applied voter set.

For example:

size = 6
target = 5

DELVOTER A
    pre-validation sees size = 6

DELVOTER B
    pre-validation also sees size = 6

after both apply:

    6 -> 5 -> 4

To prevent this, Cluster V2 now allows at most one distinct
quorum-changing membership transition to remain unapplied at a time.

The following entries are considered quorum-changing:

• ADD_VOTER
• DEL_VOTER
• NODE_JOIN ... voter
• NODE_FORGET of a voter

Learner admission itself (NODE_JOIN ... learner) does not modify the voting
quorum and is therefore not serialized by this rule.

A membership transition is identified by:

(entry type, target node id)

The leader examines the unapplied log suffix:

(last_applied, raftLogLastIndex()]

and classifies a candidate transition as:

• NONE — no quorum-changing transition is currently in flight;
• SAME — the same transition is already present in the log;
• CONFLICT — a different quorum-changing transition is still unapplied.

A CONFLICT transition is rejected.

For a retried pending proposal, SAME means the pending proposal remains
associated with the already existing log entry instead of appending another
copy.

This is particularly important across leader changes: peer match_index
values are reset when a new leader is elected, so re-running learner readiness
validation for an already logged ADD_VOTER could incorrectly reject a
transition that is already being replicated.

This serialization rule applies to both automatic and manual membership
operations, not only to the auto-promoter.

Replacement-first voter changes

Because the automatic target is currently fixed at 5, manually reducing the
voter set below that target would conflict with the controller's policy.

CLUSTER DELVOTER therefore rejects a demotion that would reduce the voter
count below RAFT_TARGET_VOTERS.

Forgetting a voting node is subject to the same rule.

To replace a voter, the administrator uses a replacement-first workflow:

5 voters

    CLUSTER ADDVOTER <new>

6 voters

    CLUSTER DELVOTER <old>

5 voters

A manual ADDVOTER may temporarily increase the voter count above the target.
The controller does not automatically demote the excess voter.

A future configurable voter target can provide the mechanism for explicitly
shrinking the voting set.

Relationship to future Cluster V2 work

Separating node admission from voting membership also helps future workflows
such as non-singleton cluster merging (#3868).

Joining nodes can first be admitted as learners without changing the voting
quorum, then promoted one at a time until the voter target is reached.

Candidate selection is intentionally simple in this PR. More advanced
placement policies, such as availability-zone-aware voter selection, belong
to #4393.

Safety scope

This PR serializes distinct quorum-changing membership transitions, but it
does not replace Cluster V2's existing config-on-apply membership model.

In particular, it does not implement Ongaro-style config-on-append or joint
consensus, and it does not eliminate the existing config-on-apply quorum-view
limitation documented in design-docs/cluster-raft.md.

The serialization rule follows the same engineering principle as Raft's
single-server membership changes while retaining Cluster V2's current
config-on-apply semantics.

Non-goals

Deferred to follow-up work:

• configurable / replicated voter target;
• automatic voter demotion;
• automatic failed-voter replacement;
• availability-zone-aware voter selection (Raft Cluster: Balance voters across availability zones #4393);
• chained learner replication (Raft Cluster: Chained learner replication #4394);
• config-on-append or joint consensus.

Test cleanup

The test-only raft_promote_start_cluster_voters bridge and most explicit
raft_add_voter calls are no longer needed because ordinary Raft cluster
startup now converges automatically.

The PR also consolidates Raft test helpers so wire-protocol tests and
higher-level cluster tests reuse the same support code.

Tests

tests/unit/cluster/cluster-raft-autopromote.tcl covers:

• automatic convergence from 1 voter + 4 learners to 5 voters;
• clusters with fewer nodes than the target;
• stopping exactly at the target;
• manual ADDVOTER above the target without automatic demotion;
• replacement-first DELVOTER;
• rejection of voter removal below the target;
• failed voters not triggering automatic replacement;
• learners that have not caught up;
• serialization of distinct membership transitions;
• leader changes;
• concurrent DELVOTER operations not reducing the voter set below target.

Existing Raft suites and generic cluster tests under --cluster-raft were
also updated to rely on automatic voter promotion instead of test-only manual
promotion.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0058145d-8f4d-4801-b5e7-0bc28f102975

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[valkey-review-bot]

Found two membership-change paths that can violate the target or duplicate an in-flight transition.

[codecov]

Codecov Report

❌ Patch coverage is 73.25581% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.55%. Comparing base (f278b08) to head (fa3c838).

Files with missing lines	Patch %	Lines
src/cluster_raft.c	73.25%	23 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff               @@
##           cluster-v2    #4456      +/-   ##
==============================================
+ Coverage       76.49%   76.55%   +0.05%     
==============================================
  Files             166      166              
  Lines           83713    83789      +76     
==============================================
+ Hits            64037    64142     +105     
+ Misses          19676    19647      -29     

Files with missing lines	Coverage Δ
src/cluster_raft.c	70.53% <73.25%> (+1.62%)	⬆️

... and 22 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.