Have kdrant migrate create the collection it migrates into - #116
Merged
Conversation
The first release build caught it: the migration refused because the target did not exist, which is true of every collection nobody has made yet and therefore of the whole command. The workflow builds the binaries and proves them against a real Qdrant before anything is published, so nothing shipped. The target is created from the source's own vectors rather than from a size and a distance the person at the terminal did not choose. --shards and --replicas override, which is what turns a copy into the re-shard the help text was already promising. The decision is a function of its inputs rather than a lambda that writes into a builder, so it is asserted without a Qdrant to write to: sizes, distances, storage, named vectors, a multi-vector comparator and a sparse modifier all have to arrive intact. HNSW, quantization and optimizer settings deliberately do not, because they were tuned for a layout the target may not have.
|
This pull request adds no capability the code did not already have. 4 files read · 3227 ms · against |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
v2.2.0tag was pushed and the release stopped where it was supposed to stop.kdrant migrate cli-source cli-targetfailed with the target collection 'cli-target' does not exist,which is true of every collection nobody has made yet and therefore of the whole command. Nothing was
published: the
publishjob hasneeds: cli, so the binaries are built and proven against a realQdrant before any jar reaches Maven Central. There is no
2.2.0on Central and no GitHub Release.The fix
kdrant migratecreates the target from the source's own vectors. Asking the person at a terminal torestate a vector size and a distance they did not choose is asking them to get one wrong, and a
migration that does not re-embed keeps the same vectors by definition.
--shardsand--replicasoverride, which is what turns a copy into the re-shard the help text was already promising.
What is deliberately not carried across is HNSW tuning, quantization and optimizer settings. Those were
changed on the source for a layout the target may not have, and copying them onto a collection that is
about to be re-sharded carries a decision made for something else.
Why it took a release build to find it
Because nothing below that level could.
ArgumentsTestcovers the flags, and the flags were right. Thekdrant-climodule has no integration test of its own, so the first thing that ran the command end toend was the release workflow, which is the last line of defence and should not have been the first.
So the decision is now a function of its inputs rather than a lambda that writes into a builder, and
MigrateTargetTestasserts it without needing a Qdrant to write to: sizes, distances, on-disk storage,named vectors, a multi-vector comparator and a sparse
IDFmodifier all have to arrive intact, and thetuning has to not.
After this merges
The
v2.2.0tag points at a commit without this fix and published nothing, so it moves to the new mergecommit rather than becoming a
2.2.1for a release that never existed. The changelog entry for M49 sayswhat the command does now.