feat(index): support distributed LABEL_LIST index - #5262
Conversation
9c2d6ba to
62799fb
Compare
| assert indices["nested_text_idx"].field_names == ["meta.text"] | ||
| assert indices["literal_dot_text_idx"].field_names == ["meta.`a.b`"] | ||
| assert indices["hyphen_user_id_idx"].field_names == ["`meta-data`.`user-id`"] | ||
| assert indices["hyphen_user_id_idx"].field_names == ["meta-data.user-id"] |
There was a problem hiding this comment.
Must we remove the ` character?
There was a problem hiding this comment.
Yes, for this assertion. This change comes from upstream: lance-format/lance#7503. describe_indices() now uses minimal field-path quoting, so hyphens no longer add backticks. With the updated Lance version, the returned field name is meta-data.user-id, so the test expectation needs to match it.
| case "LABEL_LIST": | ||
| if not ( | ||
| pa.types.is_list(field.type) or pa.types.is_large_list(field.type) | ||
| ): | ||
| raise TypeError( | ||
| f"Column {column} must be list or large list type for " | ||
| f"LABEL_LIST index, got {field.type}" | ||
| ) |
There was a problem hiding this comment.
The newly added LABEL_LIST type validation is only executed when index_type is a string. If the caller passes in an IndexConfig object in accordance with the public contract, this early validation will be skipped. As a result, invalid column types cannot be detected in advance on the Driver side, and failures may be deferred until the Ray worker execution phase. right?
There was a problem hiding this comment.
Yes, that’s right. The existing driver-side checks for the other scalar indexes also only run when index_type is a string; IndexConfig skips them today. This PR follows the existing pattern and keeps the change focused on adding LABEL_LIST. I’m considering aligning validation for string and IndexConfig inputs in lance-format/lance-ray#5250, rather than mixing that broader change into this PR.
2 participants
Summary
LABEL_LISTbuilds through Lance segment APIs (create_index_uncommittedandcommit_existing_index_segments)ScalarIndexQueryLABEL_LISTas a distributed scalar indexWhy
PyLance v10.0.0-beta.7 exposes the existing LABEL_LIST segment lifecycle through
create_index_uncommitted, including lance-format/lance#7884. Lance-Ray otherwise rejectsLABEL_LISTas a distributed index and leaves it on the legacy fragment-metadata path.Dependency
pylance>=10.0.0b7Validation
.venv/bin/pytest -q tests/test_vector_index_options.py tests/test_distributed_indexing.py::TestDistributedLabelListIndexing— 16 passed.venv/bin/python -c "import lance; print(lance.__version__); print(lance.LanceDataset._is_segment_native_scalar_index_type(\"LABEL_LIST\"))"—10.0.0-beta.7,True.venv/bin/ruff check --no-cache lance_ray tests.venv/bin/ruff format --no-cache --check lance_ray testsuv lock --checkgit diff --check