refactor: record and drive shard splits from the parent shard controller - #1328
Closed
mattisonchao wants to merge 1 commit into
Closed
mattisonchao wants to merge 1 commit into
mattisonchao wants to merge 1 commit into
Conversation
A split used to be started by the coordinator, which wrote the parent and both children through UpdateNamespaceStatus: that replaces the whole namespace blob, so a shard controller writing its own shard between the coordinator's read and its store lost that write. Splits are now recorded by a single metadata operation, InitShardSplit, which marks the parent and creates both children in one update and checks its preconditions inside the CAS. Only the values the metadata layer cannot derive are passed in: the reserved child ids, the split point and the children's ensembles. The children's hash ranges and the rest of their metadata are built there, so a child cannot disagree with its parent. The split round itself becomes Splitting, owned by the parent's shard controller the way Election already is: created from a Split action or resumed from persisted state at startup, held as currentSplitting, and stopped with the controller. The phases still run in SplitController for now; they move onto the round in a later change. The coordinator keeps what needs the cluster-wide view — validating, reserving the child ids and placing the children — and hands the rest to the parent's controller.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
runtime.InitiateSplitrecorded a split withUpdateNamespaceStatus, which replaces the whole namespace blob. Its read-modify-write happens outside the CAS, so any shard controller that wrote its own shard in between lost that write:The split then kept writing the parent and both children from its own goroutine, racing the parent controller's elections on the same shard.
Change
Metadata.InitShardSplit— one operation that marks the parent and creates both children in a single update, with its preconditions checked inside the CAS. Only what the metadata layer cannot derive is passed in (reserved child ids, split point, the children's ensembles); the children's hash ranges and the rest of their metadata are built there, so a child cannot disagree with the parent it was split from.Splitting— one round of splitting a shard, owned by the parent's shard controller the wayElectionalready is: created from aSplitaction or resumed from persisted state at startup, held ascurrentSplitting, stopped with the controller.runtime.splitControllersandrestartInProgressSplitsare gone.The coordinator keeps what needs the cluster-wide view — validation,
ReserveShardIDs, ensemble placement — and dispatches the rest. The action is dispatched without the coordinator lock, since it is completed by the controller's event loop, whose other work (LeaderElected) calls back into the coordinator.The split phases still run in
SplitControlleron its own goroutine. They move onto the round in a follow-up, which is when the remaining split writes come into the loop.Test plan
go test ./coordinator/... -count=1(oxiad)go test ./coordinator/ -run 'TestCoordinator_ShardSplit|TestCoordinator_AutoSplit'(tests) — 148s, all passInitShardSplitcoverage: children partition the parent's range, parent keeps serving until complete, every precondition rejects and persists nothing, caller ensembles stay isolatedmake lint— could not run locally: golangci-lint is built with go1.26, config targets 1.27Follow-ups
CompleteShardSplit/AbortShardSplit, then the phases move ontoSplittingand the split's writes move into the controller's event loop.