Skip to content

feat: add support for PolarDB for PostgreSQL HNSW - #860

Open
cxyybless wants to merge 1 commit into
zilliztech:mainfrom
cxyybless:feat/polardb-pg-hnsw
Open

feat: add support for PolarDB for PostgreSQL HNSW#860
cxyybless wants to merge 1 commit into
zilliztech:mainfrom
cxyybless:feat/polardb-pg-hnsw

Conversation

@cxyybless

Copy link
Copy Markdown

Summary

This PR adds a pgvector-compatible PolarDB for PostgreSQL HNSW client
to VectorDBBench. It is separate from the existing MySQL-compatible
PolarDB client and does not change the existing polardb* commands.

Supported features

  • HNSW index creation and search
  • PQ, SQ4, SQ8, and RaBitQ internal quantization
  • CLI command: vectordbbench polardbpghnsw
  • Streamlit configuration and result visualization
  • Optional Graph Cache readiness handling before search benchmarks
  • Reuse of the existing pgvector dependency extra

PolarDB-specific internal quantization and Graph Cache options require
PolarDB vector extension 0.8.3.1 or later.

Implementation

  • Reuse the PgVector client and HNSW configuration where behavior is
    compatible.
  • Add PolarDB-specific index options and validation without duplicating
    the PostgreSQL client implementation.
  • Build or validate Graph Cache after index creation and for search-only
    runs when requested.
  • Redact PostgreSQL connection passwords from client logs.
  • Add PolarDBPG configuration, lifecycle, CLI, and frontend tests.

Testing

  • Python 3.11.15:
    • make lint
    • existing dataset unit test
    • 76 PolarDBPG and client-resolution tests
  • Python 3.12.13:
    • make lint
    • existing dataset unit test
    • 76 PolarDBPG and client-resolution tests
    • wheel build with pip wheel --no-deps .
  • PolarDB for PostgreSQL 16.14 with vector extension 0.8.3.1:
    • full 50K x 1536D load and RaBitQ 8-bit HNSW index build
    • Graph Cache reached ready and usable=yes before search
    • Recall@100: 0.9909
    • search-only runs with Graph Cache lifecycle enabled and skipped
    • plain, PQ, SQ4, and SQ8 HNSW index creation and query smoke tests
    • verified that logs and result files do not expose the database password

Add a pgvector-compatible PolarDB for PostgreSQL HNSW client to the
command-line and Streamlit interfaces. Keep it separate from the existing
MySQL-compatible PolarDB client.

Support PQ, SQ4, SQ8, and RaBitQ index options and optional Graph Cache
readiness handling. Reuse the pgvector dependency extra, add configuration
and lifecycle tests, and redact connection passwords from PgVector logs.
@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cxyybless
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cxyybless

Copy link
Copy Markdown
Author

/assign @XuanYang-cn

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants