[Feature] Add Distributed Posting Router for SPANN - #448

Open

Terrence (TerrenceZhangX) wants to merge 65 commits into

users/qiazh/pre-merge-tikv-bugfixfrom

users/zhangt/merge-distributed-to-tikv

Commits on May 20, 2026

Replay distributed work onto users/qiazh/pre-merge-tikv-bugfix

authored and
Terrence (TerrenceZhangX)
committed
8716007
Fix unneede diff
Terrence (TerrenceZhangX)
committed
4186747
Remove unused stride-shard experiment

Terrence (TerrenceZhangX)
and
Copilot
committed
ee97d3f
InsertVectors: dedupe branches, log InsertThreadNum ignore in bulk path

Terrence (TerrenceZhangX)
and
Copilot
committed
4df704f
Restore (layers+1) multiplier in BlockController IO queue size

Terrence (TerrenceZhangX)
and
Copilot
committed
c27a109
SetVersionBatch: bypass LRU cache, read TiKV directly

Terrence (TerrenceZhangX)
and
Copilot
committed
f3a9de9
Drop high-priority job queue from SPDKThreadPool

Terrence (TerrenceZhangX)
and
Copilot
committed
f35ae85
Fix space
Terrence (TerrenceZhangX)
committed
a49b26d
Fix distributed benchmark README + drop dead orchestrator code

Terrence (TerrenceZhangX)
and
Copilot
committed
689e5b2
README: clarify driver = worker 0 + dispatcher; workers peer-to-peer

Terrence (TerrenceZhangX)
and
Copilot
committed
ee405d4
README: drop unused TiKV pre-split helper section

Terrence (TerrenceZhangX)
and
Copilot
committed
6cf7d36
Clean comment
Terrence (TerrenceZhangX)
committed
07bdc03

Commits on May 21, 2026

Extract IsRemoteOwnedHead predicate for owner-ring checks

Terrence (TerrenceZhangX)
and
Copilot
committed
f0d8fe5
VersionMap extend: use stride formula capacity*numWorkers

Terrence (TerrenceZhangX)
and
Copilot
committed
d55de54
RemotePostingOps: move RPC chunk/retry/timeout/inflight into INI options

Terrence (TerrenceZhangX)
and
Copilot
committed
3703866
Async Split/Merge jobs: retry counter + re-enqueue on failure

Terrence (TerrenceZhangX)
and
Copilot
committed
9619b2f
DispatchResult: carry SPTAG::ErrorCode back to driver

Terrence (TerrenceZhangX)
and
Copilot
committed
864e268
AppendCallback: HandleRaceCondition gate against in-flight split/merge

Terrence (TerrenceZhangX)
and
Copilot
committed
1cd19f1
SPANN distributed: TTL-based remote lock lease

Terrence (TerrenceZhangX)
and
Copilot
committed
dca197b
SPANN distributed: watchdog for failed async append batches

Terrence (TerrenceZhangX)
and
Copilot
committed
489ff4e
SPANN distributed: durable HeadSync log + Split WAL scaffolding

Terrence (TerrenceZhangX)
and
Copilot
committed
7093d40
SPANN distributed: full lease-fencing with monotonic fencing tokens

Terrence (TerrenceZhangX)
and
Copilot
committed
111d37c
SPANN distributed: wire split path through fenced cross-owner write

Terrence (TerrenceZhangX)
and
Copilot
committed
74c0350
SPANN distributed: route inner layers, retire async-job UAF, larger RPC chunks, remote stats

Terrence (TerrenceZhangX)
and
Copilot
committed
de3fa64
feat(distributed): receiver-side durable Batch WAL for RemoteAppend (Option A)

Terrence (TerrenceZhangX)
and
Copilot
committed
06d8899
fix(distributed): receiver-side admission control for Batch WAL

Terrence (TerrenceZhangX)
and
Copilot
committed
7aca9f0

Commits on May 22, 2026

Commits on May 24, 2026

Commits on May 25, 2026

Remove unused variable
Terrence (TerrenceZhangX)
committed
17e8646

Commits on May 27, 2026

fix(socket): typo in SimpleSerialization static_assert messages

Terrence (TerrenceZhangX)
and
Copilot
committed
82dc35a
build(test): gate absl_* link deps behind if(TIKV)

Terrence (TerrenceZhangX)
and
Copilot
committed
8fd4c30
fix(socket): separate error_code per endpoint() call in Connection

Terrence (TerrenceZhangX)
and
Copilot
committed
047ed5b
fix(distributed): explicit field-wise Encode/Decode for SplitWAL::Record

Terrence (TerrenceZhangX)
and
Copilot
committed
1786092
fix(distributed): graceful WorkerNode shutdown drains auto-flush threads

Terrence (TerrenceZhangX)
and
Copilot
committed
149bdd4
fix(bench): move SPTAGTest CWD to per-scale scratch dir on NVMe

Terrence (TerrenceZhangX)
and
Copilot
committed
5d4dbd3

Commits on May 30, 2026

Merge users/qiazh/pre-merge-tikv-bugfix: hashmap LocalVersionMap + CAS

Terrence (TerrenceZhangX)
and
Copilot
committed
8ccbd98

Commits on May 31, 2026

Commits on Jun 8, 2026

Merge remote-tracking branch 'origin/users/qiazh/pre-merge-tikv-bugfix' into users/zhangt/merge-distributed-to-tikv
Terrence (TerrenceZhangX)
committed
4768ca3

Commits on Jun 16, 2026

Commits on Jun 25, 2026

Merge branch 'users/qiazh/pre-merge-tikv-bugfix' into users/zhangt/merge-distributed-to-tikv

Terrence (TerrenceZhangX)
and
Copilot
committed
7856010

Commits on Jul 13, 2026

Commits on Jul 31, 2026

update some modification to remove old version codes.
Qi Chen
committed
b04ffa7
Merge branch 'users/zhangt/merge-distributed-to-tikv' of https://github.com/microsoft/SPTAG into users/zhangt/merge-distributed-to-tikv
Qi Chen
committed
9e42050

Commits on Aug 1, 2026

Fix distributed VID bounds and head synchronization

Terrence (TerrenceZhangX)
and
Copilot
committed
c2c6868
Merge target branch and persist distributed VID base

Terrence (TerrenceZhangX)
and
Copilot
committed
41e33aa
Drain distributed append queues before shutdown

Terrence (TerrenceZhangX)
and
Copilot
committed
f407a16

Commits on Aug 2, 2026

Merge remote distributed cleanup

Terrence (TerrenceZhangX)
and
Copilot
committed
9229f0b

Commits on Aug 3, 2026