feat: support LAION large TopK benchmarks

README.md

@@ -943,6 +943,47 @@ We've developed lots of comprehensive benchmark cases to test vector databases'
 - **Large Dataset:** Similar to the XLarge Dataset case, but uses a slightly smaller dataset (10M-1024dim, 10M-768dim, 5M-1536dim).
 - **Medium Dataset:** A case using a medium dataset (1M-1024dim, 1M-768dim, 500K-1536dim).
 - **Small Dataset:** For development (100K-768dim, 50K-1536dim).
+
+##### LAION-100M Large-TopK
+
+`Performance768D100M` selects its query and ground-truth files from `--k`:
+
+| Requested K | Query file | Ground-truth file | Queries | GT width |
+|---:|---|---|---:|---:|
+| `1..1,000` | `test.parquet` | `neighbors.parquet` | 1,000 | 1,000 |
+| `1,001..100,000` | `test_nq200.parquet` | `neighbors_top100k_nq200.parquet` | 200 | 100,000 |
+| `100,001..1,000,000` | `test_nq200.parquet` | `neighbors_top1m_nq200.parquet` | 200 | 1,000,000 |
+
+K must be positive, and LAION-100M rejects values above 1,000,000. Integer-filter runs use the smallest published GT width that covers K:
+
+| Integer filter rate | Published GT widths | Maximum K |
+|---:|---:|---:|
+| 50%, 60%, 70%, 80%, 90%, 95%, 98%, 99% | 100K, 1M | 1M |
+| 99.5% | 100K, 500K | 500K |
+| 99.8% | 100K, 200K | 200K |
+| 99.9% | 100K | 100K |
+
+K up to 1,000 keeps the original 1,000-query filtered artifacts; larger K uses `test_nq200.parquet`. Other integer filter rates and label-filter runs above K=1,000 are rejected before database initialization. VDBBench validates query IDs, row counts, and GT width before issuing a search.
+
+```bash
+vectordbbench milvusautoindex --uri http://localhost:19530 --case-type NewIntFilterPerformanceCase --dataset-with-size-type "Large LAION (768dim, 100M)" --filter-rate 0.99 --k 1000000
+```
+
+Wide GT remains in Parquet/Arrow form and is opened inside the serial-search subprocess one query row at a time. Results include primary `recall@K`, `recall_at` for the available cutoffs among 100, 1K, 10K, 100K, and 1M, plus serial and concurrent p50/p95/p99 latency. Concurrent throughput continues to use the configured fixed-duration phase.
+
+For Milvus and Zilliz Cloud performance runs with K above 16,384, VDBBench automatically creates new collections with `query_mode=large_topk` before creating the vector index. Reused collections are validated and rejected when that property is missing or incompatible. The target database, selected mode, and requested K are written to the run log. Self-hosted Milvus must be 2.6.14 or later, the release that introduced the `query_mode=large_topk` collection property; earlier servers accept the property without honoring it. Other backends must already permit the requested K; VDBBench forwards K unchanged and does not alter their collection properties.
+
+##### Performance Response Payloads
+
+Every vector search performance case can be configured for either an IDs-only response or a response that also includes each result vector. IDs only remains the default. Run the scenarios separately from the CLI:
+
+```bash
+vectordbbench milvusautoindex --uri http://localhost:19530 --case-type Performance768D100M --k 1000000 --payload-profile ids_only
+vectordbbench milvusautoindex --uri http://localhost:19530 --case-type Performance768D100M --k 1000000 --payload-profile vector
+```
+
+The frontend can select one or both scenarios for Milvus and Zilliz Cloud. Each scenario produces independent P99 latency, QPS, and recall metrics. `qps` remains the highest observed QPS among the configured concurrency levels; VDBBench does not discover a backend concurrency limit.
+
 #### Filtering Search Performance Case
 - **Int-Filter Cases:** Evaluates search performance with int-based filter expression (e.g.  "id >= 2,000").
 - **Label-Filter Cases:** Evaluates search performance with label-based filter expressions (e.g., "color == 'red'"). The test includes randomly generated labels to simulate real-world filtering scenarios.

docs/release/2026-08-large-topk.md

@@ -0,0 +1,42 @@
+# LAION-100M Large-TopK
+
+VDBBench now supports K values through 1,000,000 on the existing `Performance768D100M` case.
+
+## Dataset Selection
+
+The case selects hosted LAION artifacts from K:
+
+- K up to 1,000 uses `test.parquet` and `neighbors.parquet`.
+- K from 1,001 through 100,000 uses the 200-query `test_nq200.parquet` and `neighbors_top100k_nq200.parquet` files.
+- K from 100,001 through 1,000,000 uses `test_nq200.parquet` and `neighbors_top1m_nq200.parquet`.
+
+Integer-filter runs use their original 1,000-query artifacts through K=1,000. Above that, VDBBench uses the smallest published 200-query GT tier that covers K:
+
+| Integer filter rate | Published GT widths | Maximum K |
+|---:|---:|---:|
+| 50%, 60%, 70%, 80%, 90%, 95%, 98%, 99% | 100K, 1M | 1M |
+| 99.5% | 100K, 500K | 500K |
+| 99.8% | 100K, 200K | 200K |
+| 99.9% | 100K | 100K |
+
+The loader verifies query ID alignment, row count, and ground-truth width. Unsupported integer rates, requests above the selected filter's maximum K, label-filter runs above K=1,000, and unfiltered LAION K above 1,000,000 fail before database initialization. LAION-backed workloads that do not measure recall, such as cold latency, keep the standard 1,000-query artifacts while forwarding their configured K to the backend.
+
+## Memory And Metrics
+
+Ground truth is represented by a local Parquet path and compact metadata in the parent process. The serial-search subprocess opens that path and reads one Arrow/NumPy neighbor row at a time, avoiding conversion of the full wide GT into Python integer lists.
+
+Recall and NDCG now use O(K) hash lookups. A large-TopK serial run reports:
+
+- primary recall at the requested K;
+- `recall_at` for each supported cutoff no greater than K;
+- serial p50, p95, and p99 latency.
+
+Serial and concurrent latency fields are stored in seconds, matching the existing p95/p99 fields; the frontend converts them to milliseconds for display. `recall_at` values are ratios from 0 to 1. Existing result files load with zero/empty defaults for the new fields.
+
+## Milvus Collection Mode
+
+For Milvus and Zilliz Cloud performance runs with K above 16,384, VDBBench sets `query_mode=large_topk` when it creates the collection, before creating the vector index. The run log records the target database, the requested K, and the selected query mode. When reusing a collection, VDBBench validates the property and fails before loading or searching if the collection is incompatible.
+
+Self-hosted Milvus requires 2.6.14 or later, the release that introduced the `query_mode=large_topk` collection property. Earlier servers accept the property without honoring it, so the requested K still fails against the default 16,384 limit, and the reuse check rejects the collection on the next run.
+
+Other backends are unchanged. Their target collection must already support the requested result count before the benchmark starts.

tests/test_case_runner_reuse.py

@@ -6,6 +6,7 @@
 from vectordb_bench.backend.clients.doris.config import DorisCaseConfig, DorisConfig
 from vectordb_bench.backend.clients.pinecone.config import PineconeConfig
 from vectordb_bench.backend.clients.turbopuffer.config import TurboPufferConfig, TurboPufferIndexConfig
+from vectordb_bench.backend.clients.zilliz_cloud.config import AutoIndexConfig, ZillizCloudConfig
 from vectordb_bench.backend.data_source import DatasetSource
 from vectordb_bench.backend.dataset import DatasetWithSizeType
 from vectordb_bench.backend.task_runner import CaseRunner, RunningStatus, TaskRunner
@@ -23,6 +24,7 @@ def make_runner(
     db: DB = DB.TurboPuffer,
     db_config=None,
     db_case_config=None,
+    k: int = 100,
     stages: list[TaskStage] | None = None,
     insert_batch_size: int = DEFAULT_INSERT_BATCH_SIZE,
 ) -> CaseRunner:
@@ -33,21 +35,25 @@ def make_runner(
             db_config = PineconeConfig(api_key="key", index_name="idx")
         elif db == DB.Doris:
             db_config = DorisConfig(password=SecretStr(""))
+        elif db == DB.ZillizCloud:
+            db_config = ZillizCloudConfig(uri=SecretStr("http://example.invalid"))
         else:
             db_config = DB.Test.config_cls()
     if db_case_config is None:
         if db == DB.TurboPuffer:
             db_case_config = TurboPufferIndexConfig(metric_type=MetricType.COSINE)
         elif db == DB.Doris:
             db_case_config = DorisCaseConfig(metric_type=MetricType.COSINE)
+        elif db == DB.ZillizCloud:
+            db_case_config = AutoIndexConfig(metric_type=MetricType.COSINE)
         else:
             db_case_config = EmptyDBCaseConfig()
 
     task = TaskConfig(
         db=db,
         db_config=db_config,
         db_case_config=db_case_config,
-        case_config=CaseConfig(case_id=case_id, custom_case=custom_case or {}),
+        case_config=CaseConfig(case_id=case_id, custom_case=custom_case or {}, k=k),
         stages=stages or [TaskStage.DROP_OLD, TaskStage.LOAD, TaskStage.SEARCH_SERIAL],
         insert_batch_size=insert_batch_size,
     )
@@ -123,6 +129,13 @@ def test_reuse_key_distinguishes_insert_batch_size():
     )
 
 
+def test_reuse_key_distinguishes_zilliz_default_and_large_topk_modes():
+    assert_not_reusable(
+        make_runner(db=DB.ZillizCloud, case_id=CaseType.Performance768D100M, k=16_384),
+        make_runner(db=DB.ZillizCloud, case_id=CaseType.Performance768D100M, k=16_385),
+    )
+
+
 def test_reuse_key_distinguishes_physical_db_targets():
     assert_not_reusable(
         make_runner(db_config=TurboPufferConfig(api_key="key", region="aws-us-east-1", namespace="namespace_a")),

tests/test_cloud_payload_case.py

@@ -1,6 +1,7 @@
 import pytest
 
 from vectordb_bench import config
+from vectordb_bench.backend.assembler import Assembler
 from vectordb_bench.backend.cases import CaseType, CloudPayloadSearchCase
 from vectordb_bench.backend.clients import DB
 from vectordb_bench.backend.clients.api import EmptyDBCaseConfig
@@ -64,6 +65,52 @@ def test_case_config_builds_cloud_payload_case_from_custom_case():
     assert case.payload_profile == PayloadProfile.VECTOR
 
 
+def test_case_config_preserves_legacy_payload_profile():
+    case_config = CaseConfig(
+        case_id=CaseType.CloudPayloadSearchCase,
+        custom_case={"payload_profile": "vector"},
+    )
+
+    assert case_config.payload_profile is None
+    assert case_config.case.payload_profile == PayloadProfile.VECTOR
+
+
+def test_case_config_accepts_matching_top_level_and_legacy_payload_profiles():
+    case_config = CaseConfig(
+        case_id=CaseType.CloudPayloadSearchCase,
+        custom_case={"payload_profile": "vector"},
+        payload_profile=PayloadProfile.VECTOR,
+    )
+
+    assert case_config.payload_profile == PayloadProfile.VECTOR
+    assert case_config.case.payload_profile == PayloadProfile.VECTOR
+
+
+def test_case_config_rejects_conflicting_payload_profiles():
+    with pytest.raises(ValueError, match="conflicts with custom_case"):
+        CaseConfig(
+            case_id=CaseType.CloudPayloadSearchCase,
+            custom_case={"payload_profile": "ids_only"},
+            payload_profile=PayloadProfile.VECTOR,
+        )
+
+
+def test_assembler_preserves_top_level_performance_payload_profile():
+    task = TaskConfig(
+        db=DB.Test,
+        db_config=DB.Test.config_cls(),
+        db_case_config=EmptyDBCaseConfig(),
+        case_config=CaseConfig(
+            case_id=CaseType.Performance768D100M,
+            payload_profile=PayloadProfile.VECTOR,
+        ),
+    )
+
+    runner = Assembler.assemble("run-id", task, DatasetSource.S3)
+
+    assert runner.ca.payload_profile == PayloadProfile.VECTOR
+
+
 def test_case_runner_reuse_key_distinguishes_scalar_label_schema_requirement():
     ids_only_case = CloudPayloadSearchCase(
         dataset_with_size_type=DatasetWithSizeType.CohereSmall.value,
@@ -179,3 +226,46 @@ def test_search_runners_fail_fast_for_unsupported_payload_profile():
             k=3,
             payload_profile=PayloadProfile.VECTOR,
         )
+
+
+def test_case_runner_rejects_unsupported_payload_before_dataset_prepare(monkeypatch: pytest.MonkeyPatch):
+    events = []
+    case_config = CaseConfig(
+        case_id=CaseType.Performance768D100M,
+        payload_profile=PayloadProfile.VECTOR,
+    )
+    task = TaskConfig(
+        db=DB.Test,
+        db_config=DB.Test.config_cls(),
+        db_case_config=EmptyDBCaseConfig(),
+        case_config=case_config,
+    )
+    runner = CaseRunner(
+        run_id="run-id",
+        config=task,
+        ca=case_config.case,
+        status=RunningStatus.PENDING,
+        dataset_source=DatasetSource.S3,
+    )
+
+    monkeypatch.setattr(
+        type(runner.ca.dataset),
+        "resolve_search_files",
+        lambda self, **kwargs: events.append("resolve"),
+    )
+    monkeypatch.setattr(
+        type(runner.ca.dataset),
+        "prepare",
+        lambda self, *args, **kwargs: events.append("prepare"),
+    )
+
+    def fake_init_db(self, drop_old=True):
+        events.append("init_db")
+        self.db = FakeDB()
+
+    monkeypatch.setattr(CaseRunner, "init_db", fake_init_db)
+
+    with pytest.raises(NotImplementedError, match="payload_profile=vector"):
+        runner._pre_run(drop_old=False)
+
+    assert events == ["resolve", "init_db"]

tests/test_cloud_payload_search.py

@@ -42,6 +42,7 @@ def test_scalar_label_payload_profile_requires_scalar_label_materialization_with
 def test_dataset_prepare_loads_separated_scalar_labels_for_scalar_payload(monkeypatch):
     dataset = Dataset.LAION.manager(100_000_000)
     dataset.data.with_remote_resource = False
+    dataset.data.with_gt = False
     loaded_scalar_labels = object()
 
     def fake_read_file(file_name):
@@ -118,11 +119,13 @@ def test_serial_search_runner_passes_tenant_and_skips_recall():
         measure_recall=False,
     )
 
-    recall, ndcg, p99, p95 = runner.search((runner.test_data, runner.ground_truth))
+    recall, ndcg, p99, p95, p50, recall_at = runner.search((runner.test_data, runner.ground_truth))
 
     assert recall == 0
     assert ndcg == 0
     assert p99 >= 0
     assert p95 >= 0
+    assert p50 >= 0
+    assert recall_at == {}
     assert set(db.tenants).issubset({"tenant_0000", "tenant_0001"})
     assert db.tenants