Skip to content

refactor: record and drive shard splits from the parent shard controller - #1328

Closed
mattisonchao wants to merge 1 commit into
oxia-db:mainfrom
mattisonchao:feat/metadata-init-shard-split
Closed

mattisonchao wants to merge 1 commit into
oxia-db:mainfrom
mattisonchao:feat/metadata-init-shard-split

Conversation

@mattisonchao

Copy link
Copy Markdown
Member

Problem

runtime.InitiateSplit recorded a split with UpdateNamespaceStatus, which replaces the whole namespace blob. Its read-modify-write happens outside the CAS, so any shard controller that wrote its own shard in between lost that write:

coordinator: read namespace status ............................ store all shards
shard 7 controller:      elect leader, write shard 7 ^                      |
                                                     lost ------------------+

The split then kept writing the parent and both children from its own goroutine, racing the parent controller's elections on the same shard.

Change

Metadata.InitShardSplit — one operation that marks the parent and creates both children in a single update, with its preconditions checked inside the CAS. Only what the metadata layer cannot derive is passed in (reserved child ids, split point, the children's ensembles); the children's hash ranges and the rest of their metadata are built there, so a child cannot disagree with the parent it was split from.

Splitting — one round of splitting a shard, owned by the parent's shard controller the way Election already is: created from a Split action or resumed from persisted state at startup, held as currentSplitting, stopped with the controller. runtime.splitControllers and restartInProgressSplits are gone.

The coordinator keeps what needs the cluster-wide view — validation, ReserveShardIDs, ensemble placement — and dispatches the rest. The action is dispatched without the coordinator lock, since it is completed by the controller's event loop, whose other work (LeaderElected) calls back into the coordinator.

The split phases still run in SplitController on its own goroutine. They move onto the round in a follow-up, which is when the remaining split writes come into the loop.

Test plan

  • go test ./coordinator/... -count=1 (oxiad)
  • go test ./coordinator/ -run 'TestCoordinator_ShardSplit|TestCoordinator_AutoSplit' (tests) — 148s, all pass
  • New InitShardSplit coverage: children partition the parent's range, parent keeps serving until complete, every precondition rejects and persists nothing, caller ensembles stay isolated
  • make lint — could not run locally: golangci-lint is built with go1.26, config targets 1.27

Follow-ups

CompleteShardSplit / AbortShardSplit, then the phases move onto Splitting and the split's writes move into the controller's event loop.

A split used to be started by the coordinator, which wrote the parent and
both children through UpdateNamespaceStatus: that replaces the whole
namespace blob, so a shard controller writing its own shard between the
coordinator's read and its store lost that write.

Splits are now recorded by a single metadata operation, InitShardSplit,
which marks the parent and creates both children in one update and checks
its preconditions inside the CAS. Only the values the metadata layer cannot
derive are passed in: the reserved child ids, the split point and the
children's ensembles. The children's hash ranges and the rest of their
metadata are built there, so a child cannot disagree with its parent.

The split round itself becomes Splitting, owned by the parent's shard
controller the way Election already is: created from a Split action or
resumed from persisted state at startup, held as currentSplitting, and
stopped with the controller. The phases still run in SplitController for
now; they move onto the round in a later change.

The coordinator keeps what needs the cluster-wide view — validating,
reserving the child ids and placing the children — and hands the rest to
the parent's controller.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant