feat: support LAION large TopK benchmarks

[jamesgao-jpg]

Related to #826.

Summary

• Extend the existing LAION-100M performance case to support K through 1,000,000, including published integer-filter ground-truth tiers for supported filter rates.
• Keep wide ground truth path-based and process one query row at a time, with O(K) recall/NDCG scoring, multi-level recall_at, and p50/p95/p99 latency reporting.
• Automatically configure and validate query_mode=large_topk for Milvus and Zilliz Cloud collections when K exceeds 16,384.
• Add IDs-only and vector-payload response profiles to vector performance cases, with CLI and frontend configuration and distinct result identities.
• Propagate large-TopK settings and metrics through task execution, REST output, result compatibility, tables, and charts, including bounded frontend K controls for filtered LAION cases.
• Declare Matplotlib as a frontend runtime dependency and use its supported pyplot colormap API so clean installations can load the QPS/Recall page.

Included FTS fix

For convenience, this PR also includes a small Milvus FTS bug fix that was omitted from the FTS branch: the text field keeps its analyzer enabled without creating the unused text-match index.

Filter limitations

• Large-TopK (K > 1,000) ground truth is currently published only for NewIntFilter integer-threshold cases. LabelFilterPerformanceCase string-scalar equality remains limited to the standard LAION widths (K <= 1,000) and is rejected for large-K runs until corresponding label-filter ground-truth assets are published.

Scope

This PR contains the VDBBench implementation only. It does not publish benchmark results or product comparisons. Results and analysis will be published in follow-up PRs.

[frankleaf]

#836

Please take a look at this PR to see if it's needed.

[frankleaf]

@jamesgao-jpg
Could you please let me know if this branch is ready for testing now, and if there's any remaining development work?

[jamesgao-jpg]

@jamesgao-jpg Could you please let me know if this branch is ready for testing now, and if there's any remaining development work?

@frankleaf It should be ready for testing, and I did some preliminary testing on Zilliz Cloud already and got some results (not published yet), and the end to end result looks reasonable.

[frankleaf]

@jamesgao-jpg Sure, I'll give it a try as well. By the way, could you let me know when the Zilliz test results will be available? I'd like to take a look and compare the performance differences between Zilliz and open-source Milvus.

[frankleaf]

@jamesgao-jpg Also, I have another question: Is the current 1M GT obtained by performing FLAT brute-force search on the LAION 100M dataset?

[jamesgao-jpg]

@jamesgao-jpg Also, I have another question: Is the current 1M GT obtained by performing FLAT brute-force search on the LAION 100M dataset?

@frankleaf Conceptually yes. In practice we used GPU to parallelize computation and did it streamingly to avoid memory problem, so not exactly flat brute force in CPU as in faiss. I will upload the gt generation script later to the bucket for reproductibility.

[frankleaf]

@jamesgao-jpg I've tested it, and the 1M recall is working perfectly—great job! Also, I was wondering if there are any plans to provide a 1M test set with filters along with ground truth?

[jamesgao-jpg]

@jamesgao-jpg I've tested it, and the 1M recall is working perfectly—great job! Also, I was wondering if there are any plans to provide a 1M test set with filters along with ground truth?

@frankleaf Yes that will be the plan next. I will pin you once that's released.

[frankleaf]

@jamesgao-jpg thanks! Also, will we be launching a separate leaderboard for the Top1M scenario on https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch ?

[jamesgao-jpg]

@jamesgao-jpg thanks! Also, will we be launching a separate leaderboard for the Top1M scenario on https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch ?

@frankleaf Yes that will be the plan, but might not be released as soon, since we need to decide what's the best leaderboard format to introduce this and what backends can provide meaningful results (many backends support very limited topks right out of the box).

[frankleaf]

@jamesgao-jpg May I ask another question: could we provide datasets larger than LAION-100M to support testing scenarios for massive-scale disk index or large-top-k performance, such as datasets at the 1B or 10B scale?

[xiaofan-luan]

@jamesgao-jpg May I ask another question: could we provide datasets larger than LAION-100M to support testing scenarios for massive-scale disk index or large-top-k performance, such as datasets at the 1B or 10B scale?

this is not the goal for this benchmark. because maintain such a dataset is too costly. and also not much people will have enough resources to run the benchmark.

We will run some test in house and share the benchmark and this won't be part of the open source bench.

[frankleaf]

@xiaofan-luan get it ,thank you

[frankleaf]

@jamesgao-jpg Sorry to bother you，When will the support for large top-k be merged into the main branch? Also, when will the case with filter mentioned above be available for testing?

[jamesgao-jpg]

@jamesgao-jpg Sorry to bother you，When will the support for large top-k be merged into the main branch? Also, when will the case with filter mentioned above be available for testing?

Sorry a bit swamped last week. We will start reviewing the code and get it merged soon. Hopefully we can get it done by the end of this week (can't guarantee though). For filtered cases we will generate filtered gts ASAP (havn't gotten time to do it yet :(

[frankleaf]

@jamesgao-jpg Sorry to bother you，When will the support for large top-k be merged into the main branch? Also, when will the case with filter mentioned above be available for testing?

Sorry a bit swamped last week. We will start reviewing the code and get it merged soon. Hopefully we can get it done by the end of this week (can't guarantee though). For filtered cases we will generate filtered gts ASAP (havn't gotten time to do it yet :(

thank you! Please ping me when it's ready

[frankleaf]

@jamesgao-jpg by the way, I've submitted a PR to make the segment size for force merge configurable during Milvus testing. The current default size lacks flexibility in real-world testing scenarios, particularly due to memory constraints on DataNodes and QueryNodes (governed by queryNodeMemoryFactor and dataNodeMemoryFactor). I'd like to be able to adjust the segment size flexibly in practice, while keeping the fixed maximum value as the default. Could you please review whether this parameter can be supported?

#861

[jamesgao-jpg]

Review: LargeTopk -> main (PR #834)

Reviewed origin/main...LargeTopk at head a90f347 (re-reviewed after the head moved 8969f09 -> a90f347; the only delta is docs-removal commit a90f347, no source changes, findings unchanged). 36 files, +3308/-149, DCO sign-off on all commits, git diff --check clean.

No P0/P1/P2 findings. Two P3 items posted inline below.

Verified solid: K propagation (CLI IntRange(min=1), CaseConfig.validate_k, requested_k or K_DEFAULT), ground-truth tiering matches the documented table with fail-fast before DB init, ParquetGroundTruth validates query-ID alignment / row count / width and streams wide rows without materializing Python lists, load_reuse_key includes the query_mode=large_topk collection properties, new result fields are additive with read-time defaults (p50 s->ms conversion mirrors p99/p95), FTS is unaffected (separate FtsDatasetManager), p50/recall_at tuple plumbing is consistent across serial / mp / read-write / cold-warm runners, and frontend K/payload-aware case naming is consistent across filters, charts, and tables.

[jamesgao-jpg]

@frankleaf Update: the LAION-100M integer-filter large-top-K ground truth is now generated, independently verified, and publicly available.

It covers filter rates 50%, 60%, 70%, 80%, 90%, 95%, 98%, 99%, 99.5%, 99.8%, and 99.9%. Every rate has top-100K ground truth. Top-1M is available wherever at least 1M vectors remain eligible; the narrower eligible populations are capped accordingly: top-500K at 99.5%, top-200K at 99.8%, and top-100K at 99.9%.

The complete artifact inventory, sizes, and SHA-256 checksums are documented here:

• AWS: https://assets.zilliz.com/benchmark/laion_large_100m/README.md
• Aliyun: https://assets.zilliz.com.cn/benchmark/laion_large_100m/README.md

We don't think we will publish the scalar filter gts yet. The goal is to get this pr (which contains implementations only) merged. Next we will gradually roll out scalar filter and maybe have benchmark results in upcoming prs.

[pymilvus-bot]

[P1] Preserve the payload profile for custom-dataset performance cases

CaseConfig.case injects the top-level payload_profile into the case constructor, but PerformanceCustomDataset.__init__ accepts **kwargs and does not forward them to super().__init__. As a result, a custom-dataset configuration with payload_profile=vector silently instantiates an ids_only case.

This runs the wrong workload and can leave the serialized CaseConfig (vector) inconsistent with the recorded metrics (ids_only); IDs-only and vector results may also be merged under the same result identity.

Please forward **kwargs in the super().__init__ call and add a regression test covering CaseConfig(case_id=PerformanceCustomDataset, payload_profile=PayloadProfile.VECTOR).case.payload_profile.

[XuanYang-cn]

/approve
LGTM, this PR can be merged after the pymilvus-bot's review comments been addressed.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jamesgao-jpg, XuanYang-cn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [XuanYang-cn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jamesgao-jpg]

Addressed the custom-dataset payload forwarding issue in #862. The fix forwards **kwargs to PerformanceCase and adds a regression test proving that CaseConfig(case_id=PerformanceCustomDataset, payload_profile=vector) produces a vector-payload case.

I also audited the other registered PerformanceCase constructors affected by the top-level payload argument. The standard cases inherit the base initializer, and NewIntFilterPerformanceCase, CloudPayloadSearchCase, CloudMultiTenantSearchCase, and LabelFilterPerformanceCase already forward **kwargs; no additional forwarding gaps were found.