Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -76,13 +76,13 @@ RUN --mount=type=cache,target=/root/.cache/uv,id=uv-${TARGETPLATFORM} \
fi; \
case "${UV_LOCK_STRATEGY}" in \
locked) \
uv sync --locked --no-editable --extra bot --extra gemini \
uv sync --locked --no-editable --extra bot --extra gemini --extra opengauss \
;; \
auto) \
if ! uv lock --check; then \
uv lock; \
fi; \
uv sync --locked --no-editable --extra bot --extra gemini \
uv sync --locked --no-editable --extra bot --extra gemini --extra opengauss \
;; \
*) \
echo "Unsupported UV_LOCK_STRATEGY: ${UV_LOCK_STRATEGY}" >&2; \
Expand Down
37 changes: 36 additions & 1 deletion docs/en/guides/01-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1351,7 +1351,7 @@ Vector database storage configuration

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `backend` | str | VectorDB backend type: 'local' (file-based), 'http' (remote service), 'volcengine' (cloud VikingDB), 'vikingdb' (private deployment), or 'cuvs' (local storage + GPU dense search) | "local" |
| `backend` | str | VectorDB backend type: 'local' (file-based), 'http' (remote service), 'volcengine' (cloud VikingDB), 'vikingdb' (private deployment), 'cuvs' (local storage + GPU dense search), or 'opengauss' | "local" |
| `name` | str | VectorDB collection name | "context" |
| `url` | str | Remote service URL for 'http' type (e.g., 'http://localhost:5000') | null |
| `project_name` | str | Project name (alias project) | "default" |
Expand All @@ -1361,6 +1361,7 @@ Vector database storage configuration
| `volcengine` | object | 'volcengine' type VikingDB configuration | - |
| `vikingdb` | object | 'vikingdb' type private deployment configuration | - |
| `cuvs` | object | NVIDIA cuVS configuration for the 'cuvs' backend and the opt-in memory-aware auto mode on 'local'; see the [cuVS guide](./16-cuvs.md) | - |
| `opengauss` | object | openGauss DataVec config, see [openGauss guide](./19-opengauss.md) | - |

Default local mode
```
Expand All @@ -1373,6 +1374,40 @@ Default local mode
}
```

<details>
<summary><b>openGauss</b></summary>

```json
{
"storage": {
"vectordb": {
"backend": "opengauss",
"dimension": 512,
"distance_metric": "cosine",
"opengauss": {
"host": "127.0.0.1",
"port": 5432,
"user": "gaussdb",
"password": "replace-me",
"db_name": "openviking",
"mode": "standalone",
"shard_count": 32,
"index_type": "hnsw",
"build_params": {"m": 16, "ef_construction": 64},
"search_params": {"ef_search": 100},
"parallel_workers": 4,
"connection_pool_min_size": 1,
"connection_pool_max_size": 8
}
}
}
}
```

See the [openGauss guide](./19-opengauss.md) for PQ, RabitQ, DiskANN, lifecycle, and distributed configuration.

</details>

<details>
<summary><b>volcengine vikingDB</b></summary>
Supports cloud-deployed VikingDB on Volcengine
Expand Down
79 changes: 79 additions & 0 deletions docs/en/guides/19-opengauss.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# openGauss Vector Backend

OpenViking integrates openGauss DataVec through the VectorDB Adapter. Upper-layer `find/search` calls remain unchanged; SQL, index lifecycle, pooling, and distributed behavior stay inside the adapter.

## Installation

```bash
pip install "openviking[opengauss]"
```

## First-party configuration

Configure indexes only under `storage.vectordb.opengauss` with `index_type + build_params + search_params`. `custom_params` is not the first-party openGauss configuration path.

```json
{
"storage": {
"vectordb": {
"backend": "opengauss",
"name": "context",
"dimension": 512,
"distance_metric": "cosine",
"opengauss": {
"host": "127.0.0.1",
"port": 5432,
"user": "gaussdb",
"password": "replace-me",
"db_name": "openviking",
"mode": "standalone",
"shard_count": 32,
"index_type": "hnsw-pq",
"build_params": {"m": 16, "ef_construction": 64, "pq_m": 8, "pq_ksub": 256},
"search_params": {"ef_search": 100},
"parallel_workers": 4,
"maintenance_work_mem_mb": 128,
"connection_pool_min_size": 1,
"connection_pool_max_size": 8
}
}
}
}
```

Supported logical types are `hnsw`, `hnsw-pq`, `hnsw-rabitq`, `ivfflat`, `ivf-pq`, `ivf-rabitq`, and `diskann`. Quantized variants use the base `hnsw` or `ivfflat` access method with `enable_pq=on` or `enable_rabitq=on`. DiskANN-PQ is not supported; `diskann` configurations cannot set `enable_pq` or `pq_m`.

## Build parameters

- **HNSW:** `m` 2–100; `ef_construction` 4–1000 and at least `2*m`.
- **IVFFlat:** `lists` 1–32768.
- **DiskANN:** `index_size` 16–1000.
- **PQ:** HNSW/IVF support `enable_pq`, `pq_m`, and `pq_ksub`; `dimension % pq_m == 0`. `pq_m` is 1–2000 and `pq_ksub` is 1–256. IVF-PQ also supports `by_residual`.
- **RabitQ:** `enable_rabitq`, `rabitq_refine_type` (`none`, `SQ8`, `FP32`), and `rabitq_fht`. PQ and RabitQ are mutually exclusive. DiskANN does not support RabitQ.
- **Parallel build:** top-level `parallel_workers` is 0–32.
- **Build memory:** top-level `maintenance_work_mem_mb` defaults to 64 MiB and accepts 16–1048576. The adapter applies `SET LOCAL maintenance_work_mem` only in the ANN `CREATE INDEX` transaction, without changing the database-wide setting. Increase it when IVFFlat, PQ, or larger datasets report insufficient index-build memory.

Unknown parameters and invalid combinations fail configuration validation.

## Search parameters

- HNSW: `ef_search` and `earlystop_threshold` map to `hnsw_ef_search` and `hnsw_earlystop_threshold`.
- IVF: `probes` maps to `ivfflat_probes`; IVF-PQ also supports `ivfpq_kreorder`.
- DiskANN: `probes` maps to `diskann_probes`.
- RabitQ: `rbq_query_bits` and `rbq_refinek` map to the same-named GUCs.

The adapter applies these with `SET LOCAL` in the search transaction. In distributed mode it first executes `SET LOCAL spq.propagate_set_commands = 'local'` in the same transaction, because spq defaults to `none` and would otherwise leave DN shard scans on server-default parameters.

## Lifecycle and consistency

Plain HNSW may be created on an empty table. IVFFlat, PQ, RabitQ, and DiskANN are built after data exists; a `bulk_ingest` scope builds once when the outer scope ends. Physical access method, operator class, and options are verified through the catalog before `_ov_index_*` metadata is persisted. Invalid historical metadata is removed. Search fails if the configured physical ANN index is missing or mismatched.

## Distributed mode

Set `mode=distributed` and point `host/port` to an spq CN. Startup verifies the spq extension, distribution functions, active DN workers, all-node connectivity, and catalog state in `pg_dist_partition`. Collection tables are hash distributed by `id`. Standard spqplugin_v2 hash-distributes metadata by `table_name` and `index_name`; a Citus-compatible CN may use reference tables when it explicitly provides `create_reference_table`.

The official openGauss distributed-vector example guarantees plain HNSW, and that is the only ANN type enabled by the adapter in `mode=distributed`. Real CN + 2 DN acceptance confirmed a DN `Ann Index Scan` for plain HNSW. In the same cluster, HNSW-PQ and IVF-PQ failed while SPQ attempted to build on the zero-row CN logical table; HNSW-RabitQ, IVFFlat, and IVF-RabitQ produced DN sequential scans instead of ANN scans. The adapter therefore rejects every distributed index type except plain HNSW before connecting or creating tables. Use standalone mode for PQ, RabitQ, IVF, or DiskANN indexes.

## Verification

Inspect both `pg_indexes.indexdef` and `EXPLAIN`. A real ANN verification requires an `Ann Index Scan` using the intended physical index; recall results alone do not prove index use.
37 changes: 36 additions & 1 deletion docs/zh/guides/01-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1437,7 +1437,7 @@ Redis Sentinel 分别配置数据节点和 Sentinel 的 ACL:

| 参数 | 类型 | 说明 | 默认值 |
|------|------|------|--------|
| `backend` | str | VectorDB 后端类型: 'local'(基于文件), 'http'(远程服务), 'volcengine'(云上 VikingDB), 'vikingdb'(私有部署) 'cuvs'(本地存储 + GPU dense search) | "local" |
| `backend` | str | VectorDB 后端类型: 'local'(基于文件), 'http'(远程服务), 'volcengine'(云上 VikingDB), 'vikingdb'(私有部署), 'cuvs'(本地存储 + GPU dense search)或 'opengauss' | "local" |
| `name` | str | VectorDB 的集合名称 | "context" |
| `url` | str | 'http' 类型的远程服务 URL(例如 'http://localhost:5000') | null |
| `project_name` | str | 项目名称(别名 project) | "default" |
Expand All @@ -1447,6 +1447,7 @@ Redis Sentinel 分别配置数据节点和 Sentinel 的 ACL:
| `volcengine` | object | 'volcengine' 类型的 VikingDB 配置 | - |
| `vikingdb` | object | 'vikingdb' 类型的私有部署配置 | - |
| `cuvs` | object | NVIDIA cuVS 配置,也用于在 'local' 下显式开启显存感知自动模式,参见 [cuVS 使用指南](./16-cuvs.md) | - |
| `opengauss` | object | openGauss DataVec 配置,参见 [openGauss 使用指南](./19-opengauss.md) | - |

默认使用本地模式
```
Expand All @@ -1459,6 +1460,40 @@ Redis Sentinel 分别配置数据节点和 Sentinel 的 ACL:
}
```

<details>
<summary><b>openGauss</b></summary>

```json
{
"storage": {
"vectordb": {
"backend": "opengauss",
"distance_metric": "cosine",
"opengauss": {
"host": "127.0.0.1",
"port": 5432,
"user": "gaussdb",
"password": "Gauss@123",
"db_name": "openviking",
"mode": "standalone",
"shard_count": 32,
"index_type": "hnsw",
"build_params": { "m": 16, "ef_construction": 64 },
"search_params": { "ef_search": 100 },
"parallel_workers": 4,
"connection_pool_min_size": 1,
"connection_pool_max_size": 8
}
}
}
}
```

更多索引与分布式配置见 [openGauss 使用指南](./19-opengauss.md)。

</details>


<details>
<summary><b>volcengine vikingDB</b></summary>
支持火山引擎云上部署的 VikingDB
Expand Down
120 changes: 120 additions & 0 deletions docs/zh/guides/19-opengauss.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# openGauss 向量库后端

OpenViking 通过 VectorDB Adapter 接入 openGauss DataVec。上层 `find/search` 调用方式不变;SQL、索引生命周期、连接池和分布式差异封装在 Adapter 内。建议使用包含目标 DataVec 索引能力的 openGauss 版本,并以实际数据库 catalog 和 `EXPLAIN` 验证能力。

## 安装

```bash
pip install "openviking[opengauss]"
```

## 正式配置入口

索引只能通过 `storage.vectordb.opengauss` 下的 `index_type + build_params + search_params` 配置。`custom_params` 不是正式 openGauss 参数入口。

```json
{
"storage": {
"vectordb": {
"backend": "opengauss",
"name": "context",
"dimension": 512,
"distance_metric": "cosine",
"opengauss": {
"host": "127.0.0.1",
"port": 5432,
"user": "gaussdb",
"password": "replace-me",
"db_name": "openviking",
"mode": "standalone",
"shard_count": 32,
"index_type": "hnsw-pq",
"build_params": {
"m": 16,
"ef_construction": 64,
"pq_m": 8,
"pq_ksub": 256
},
"search_params": {
"ef_search": 100
},
"parallel_workers": 4,
"maintenance_work_mem_mb": 128,
"connection_pool_min_size": 1,
"connection_pool_max_size": 8
}
}
}
}
```

逻辑索引类型映射到 openGauss 物理 access method:

| `index_type` | 物理方法 | 自动构建选项 |
|---|---|---|
| `hnsw` | `hnsw` | 无量化 |
| `hnsw-pq` | `hnsw` | `enable_pq=on` |
| `hnsw-rabitq` | `hnsw` | `enable_rabitq=on` |
| `ivfflat` | `ivfflat` | 无量化 |
| `ivf-pq` | `ivfflat` | `enable_pq=on` |
| `ivf-rabitq` | `ivfflat` | `enable_rabitq=on` |
| `diskann` | `diskann` | 无量化 |

## 构建参数

- **HNSW**:`m` 取值 2–100;`ef_construction` 取值 4–1000,且不小于 `2*m`。
- **IVFFlat**:`lists` 取值 1–32768。训练型索引在数据写入后创建。
- **DiskANN**:`index_size` 取值 16–1000。
- **PQ**:仅 HNSW/IVF 支持 `enable_pq`、`pq_m`、`pq_ksub`;`pq_m` 必须整除 `storage.vectordb.dimension`,范围为 1–2000;`pq_ksub` 为 1–256。IVF-PQ 额外支持 `by_residual`。Adapter 不支持 DiskANN-PQ,也不接受 `diskann` 搭配 `enable_pq` 或 `pq_m`。
- **RabitQ**:`enable_rabitq`、`rabitq_refine_type` (`none`/`SQ8`/`FP32`)、`rabitq_fht`。PQ 与 RabitQ 不能同时启用;DiskANN 不支持 RabitQ。
- **并行构建**:`parallel_workers` 为 0–32,Adapter 设置表的 `parallel_workers` 后构建或重建索引。
- **构建内存**:顶层 `maintenance_work_mem_mb` 默认为 64 MiB,范围 16–1048576。Adapter 仅在 ANN `CREATE INDEX` 事务内执行 `SET LOCAL maintenance_work_mem`,不会修改数据库全局配置。IVFFlat、PQ 或较大数据集若报告构建内存不足,应按 openGauss 错误提示提高该值。

未知参数、非法布尔类型和不兼容组合会在 OpenViking 启动时直接拒绝。

## 查询参数

查询参数通过同一事务内的 `SET LOCAL` 应用:

| 索引 | `search_params` | openGauss 参数 |
|---|---|---|
| HNSW 系列 | `ef_search` | `hnsw_ef_search` |
| HNSW 系列 | `earlystop_threshold` | `hnsw_earlystop_threshold` |
| IVF 系列 | `probes` | `ivfflat_probes` |
| IVF-PQ | `ivfpq_kreorder` | `ivfpq_kreorder` |
| DiskANN 系列 | `probes` | `diskann_probes` |
| RabitQ 系列 | `rbq_query_bits`, `rbq_refinek` | 同名 GUC |

兼容键 `hnsw_ef_search`、`hnsw_earlystop_threshold`、`ivfflat_probes` 和 `diskann_probes` 仍可读取。

分布式模式下,Adapter 会在同一查询事务内先执行 `SET LOCAL spq.propagate_set_commands = 'local'`:spq 默认值为 `none`,否则上述 `SET LOCAL` 查询参数只影响 CN 会话,DN 分片扫描仍使用服务端默认参数。

## 索引生命周期与一致性

- 普通 HNSW 可以在空表创建;IVFFlat、PQ、RabitQ 和 DiskANN 在数据存在后创建。
- `bulk_ingest` 期间只写数据,最外层 scope 结束后构建一次训练型索引。
- Adapter 比对物理 access method、operator class 和构建选项;参数变化时替换并重新验证索引。
- 只有 `pg_indexes` catalog 验证成功后才写 `_ov_index_*` metadata。历史假 metadata 会在加载时清理。
- 查询前目标物理 ANN 索引必须存在且与配置匹配,否则失败,不静默退化为顺序扫描。

## 分布式模式

`mode=distributed` 时 `host/port` 必须指向 openGauss spq CN。Adapter 启动时检查:

1. `spq`/`spq_plugin_v2` 扩展;
2. `create_distributed_table`;若 CN 兼容 Citus,可选使用 `create_reference_table`;
3. 至少一个 active DN worker,且 `run_command_on_all_nodes('SELECT 1')` 全部成功;
4. 业务表和 metadata 表在 `pg_dist_partition` 中存在已验证记录。

业务表按 `id` hash 分片。标准 spqplugin_v2 将 metadata 分别按 `table_name` 和 `index_name` hash 分片;只有明确提供 `create_reference_table` 的兼容集群才使用 reference table。openGauss 官方分布式向量示例明确覆盖普通 HNSW,Adapter 在 `mode=distributed` 下也只允许普通 HNSW。真实 CN + 2 DN 验收确认普通 HNSW 在 DN 使用 `Ann Index Scan`;同一集群中 HNSW-PQ 和 IVF-PQ 会在 SPQ 尝试构建零行 CN 逻辑表索引时失败,HNSW-RabitQ、IVFFlat 和 IVF-RabitQ 的 DN 计划仍为顺序扫描。Adapter 因此会在连接或建表前拒绝除普通 HNSW 之外的所有 distributed 索引类型;PQ、RabitQ、IVF 和 DiskANN 请使用 standalone。

## 验收

真实 ANN 验收必须同时检查:

```sql
SELECT indexdef FROM pg_indexes WHERE tablename = 'context';
EXPLAIN SELECT id FROM context ORDER BY vector <=> '[...]'::vector LIMIT 5;
```

执行计划应出现目标索引的 `Ann Index Scan`。仅看召回结果不能证明 ANN 索引已生效。
11 changes: 6 additions & 5 deletions openviking/storage/vectordb/collection/collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# SPDX-License-Identifier: AGPL-3.0
import importlib
from abc import ABC, abstractmethod
from typing import Any, Dict, List, Optional, Type
from typing import Any, Dict, List, Optional, Type, Union

from openviking.storage.vectordb.collection.result import AggregateResult, SearchResult
from openviking.storage.vectordb.index.index import IIndex
Expand Down Expand Up @@ -136,7 +136,7 @@ def search_by_scalar(
def update_index(
self,
index_name: str,
scalar_index: Optional[Dict[str, Any]] = None,
scalar_index: Optional[Union[List[str], Dict[str, Any]]] = None,
description: Optional[str] = None,
):
raise NotImplementedError
Expand Down Expand Up @@ -535,16 +535,17 @@ def search_by_scalar(
def update_index(
self,
index_name: str,
scalar_index: Optional[Dict[str, Any]] = None,
scalar_index: Optional[Union[List[str], Dict[str, Any]]] = None,
description: Optional[str] = None,
):
"""
Update the configuration or description of an existing index.

Args:
index_name (str): Name of the index to update.
scalar_index (Optional[Dict[str, Any]]): Updated configuration for scalar indexes.
Defaults to None.
scalar_index (Optional[Union[List[str], Dict[str, Any]]]): Updated scalar
index configuration. List-based backends use field names; legacy
backends may accept a dictionary-shaped configuration. Defaults to None.
description (Optional[str]): New description for the index. Defaults to None.
"""
if self.__collection is None:
Expand Down
6 changes: 3 additions & 3 deletions openviking/storage/vectordb/collection/http_collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# SPDX-License-Identifier: AGPL-3.0
import copy
import json
from typing import Any, Dict, List, Optional
from typing import Any, Dict, List, Optional, Union

import requests

Expand Down Expand Up @@ -217,15 +217,15 @@ def list_indexes(
def update_index(
self,
index_name: str,
scalar_index: Optional[Dict[str, Any]] = None,
scalar_index: Optional[Union[List[str], Dict[str, Any]]] = None,
description: Optional[str] = None,
):
data = {
"ProjectName": self.project_name,
"CollectionName": self.collection_name,
"IndexName": index_name,
}
if scalar_index:
if scalar_index is not None:
data["ScalarIndex"] = json.dumps(scalar_index)
if description is not None:
data["Description"] = description
Expand Down
Loading
Loading