Chunked auto-embeddings

cmake/GetColumnar.cmake

@@ -7,7 +7,7 @@ include ( update_bundle )
 # Versions of API headers we are need to build with.
 set ( NEED_COLUMNAR_API 28 )
 set ( NEED_SECONDARY_API 21 )
-set ( NEED_KNN_API 17 )
+set ( NEED_KNN_API 18 )
 
 if (WIN32)
 	set ( EXTENSION dll )

manual/english/Creating_a_table/Data_types.md

@@ -2576,6 +2576,12 @@ When creating a table with `float_vector` attributes for KNN search, you can spe
 - `HNSW_M`: Maximum connections in the graph (default: 16)
 - `HNSW_EF_CONSTRUCTION`: Construction time/accuracy trade-off (default: 200)
 
+**Chunking parameters** (when using `MODEL_NAME`, see [Chunking strategies](../Searching/KNN.md#Chunking-strategies)):
+- `CHUNK_STRATEGY`: how a document becomes vectors: `'truncate'` (default), `'mean'`, `'fixed'`, `'recursive'` or `'sentence'`. The last three produce several vectors per document and require a [`float_vector_array`](../Creating_a_table/Data_types.md#Float-vector-array) column.
+- `MAX_TOKENS`: chunk size in tokens; `0` (default) uses the model's own limit
+- `OVERLAP_TOKENS`: tokens shared between consecutive chunks; requires an explicit non-zero `MAX_TOKENS`; a large overlap is reduced so chunks still advance
+- `MAX_CHUNKS`: ceiling on vectors per document; `0` (default) means unlimited
+
 **Auto-embeddings parameters** (when using `MODEL_NAME`):
 - `MODEL_NAME`: The embedding model to use (e.g., `'Xenova/all-MiniLM-L6-v2'` for the fast ONNX path, `'sentence-transformers/all-MiniLM-L6-v2'`, or `'openai/text-embedding-ada-002'`)
 - `FROM`: Comma-separated list of field names to use for embedding generation, or empty string `''` to use all text/string fields
@@ -2957,15 +2963,15 @@ When the attribute is configured for [KNN](../Searching/KNN.md), all vectors of
 - Currently only supported in real-time tables (not in plain tables)
 - Not supported in functions or expressions
 - Cannot be used in regular filters or sorting
-- [Auto embeddings](../Searching/KNN.md#Auto-Embeddings-%28Recommended%29) are not available for this type: a model produces one vector per document, so `MODEL_NAME` is rejected. Vectors must be supplied explicitly.
+- [Auto embeddings](../Searching/KNN.md#Auto-Embeddings-%28Recommended%29) work, but only with a chunking strategy that produces several vectors per document - see [Chunking strategies](../Searching/KNN.md#Chunking-strategies). Adding a model-backed `float_vector_array` with `ALTER TABLE ... ADD COLUMN`, and `ALTER TABLE ... REBUILD EMBEDDINGS` on one, are not supported yet; declare the column when creating the table.
 - Not compatible with the [Auto schema](../Data_creation_and_modification/Adding_documents_to_a_table/Adding_documents_to_a_real-time_table.md#Auto-schema) mechanism
 
 ### Using float vector arrays with KNN
 
 The parameters are the same ones [`float_vector`](../Creating_a_table/Data_types.md#Float-vector) takes: `KNN_TYPE`, `KNN_DIMS`, `HNSW_SIMILARITY`, plus the optional `HNSW_M`, `HNSW_EF_CONSTRUCTION` and [quantization](../Searching/KNN.md#Vector-quantization), with two differences:
 
 - `KNN_DIMS` is required, and **every** vector in **every** row must have exactly that many entries. A row whose vectors are a different width is rejected on insert.
-- `MODEL_NAME` and `FROM` are not accepted.
+- `MODEL_NAME` and `FROM` are accepted, together with a multi-vector `CHUNK_STRATEGY` that fills the array automatically. See [Chunking strategies](../Searching/KNN.md#Chunking-strategies).
 
 **What you can do:**
 - Run KNN searches that match a document on its closest vector

manual/english/Creating_a_table/Local_tables/Plain_and_real-time_table_settings.md

@@ -559,8 +559,15 @@ knn = {"attrs":[{"name":"chunk_vectors","type":"hnsw","dims":768,"hnsw_similarit
 
 Two differences apply:
 
-- `dims` is **required**, and every vector in every row must have exactly that many entries.
-- `model_name` and `from` are **not** accepted — auto embeddings produce one vector per document, so they do not apply to this type. Vectors must be supplied explicitly.
+- `dims` is **required** when vectors are supplied explicitly, and every vector in every row must have exactly that many entries. It must be **omitted** when `model_name` is used.
+- `model_name`/`from` are accepted together with a multi-vector `chunk_strategy` (`fixed`, `recursive` or `sentence`), which fills the array with one vector per chunk:
+
+```ini
+rt_attr_float_vector_array = chunk_vectors
+knn = {"attrs":[{"name":"chunk_vectors","type":"hnsw","hnsw_similarity":"COSINE","model_name":"Xenova/all-MiniLM-L6-v2","from":"title,content","chunk_strategy":"sentence","max_tokens":256,"overlap_tokens":32}]}
+```
+
+  See [Chunking strategies](../../Searching/KNN.md#Chunking-strategies) for the full option list and the `ALTER` limitations.
 
 All vectors are indexed together, and a KNN search returns each document once, scored by its closest vector. See [Float vector array](../../Creating_a_table/Data_types.md#Float-vector-array) and [Multiple vectors per document](../../Searching/KNN.md#Multiple-vectors-per-document).
 

manual/english/Searching/KNN.md

@@ -604,7 +604,7 @@ The query vector is still a single vector of `KNN_DIMS` entries, exactly as for
 
 With `HNSW_SIMILARITY='cosine'`, each stored vector is normalized on its own, so a document's vectors are compared against the query individually rather than as one long concatenated vector.
 
-Everything else on this page applies unchanged: [filtering](../Searching/KNN.md#Filtering-KNN-vector-search-results), [prefilter/postfilter](../Searching/KNN.md#Filtering-strategies:-prefilter-vs.-postfilter), [quantization](../Searching/KNN.md#Vector-quantization), [early termination](../Searching/KNN.md#Early-termination) and rescoring behave the same way. The only capability that is unavailable is [auto embeddings](../Searching/KNN.md#Auto-Embeddings-%28Recommended%29), since a model yields one vector per document.
+Everything else on this page applies unchanged: [filtering](../Searching/KNN.md#Filtering-KNN-vector-search-results), [prefilter/postfilter](../Searching/KNN.md#Filtering-strategies:-prefilter-vs.-postfilter), [quantization](../Searching/KNN.md#Vector-quantization), [early termination](../Searching/KNN.md#Early-termination) and rescoring behave the same way. [Auto embeddings](../Searching/KNN.md#Auto-Embeddings-%28Recommended%29) can fill the array for you, one vector per chunk - see [Chunking strategies](../Searching/KNN.md#Chunking-strategies) below.
 
 <!-- example multi_vector -->
 
@@ -644,6 +644,77 @@ POST /search
 
 <!-- end -->
 
+### Chunking strategies
+
+By default an embedding model reads only as much of a document as fits its input window (typically a few hundred tokens) and the rest is silently dropped. For a title or a short description that is complete. For a long article it is not: nothing written past the cut-off can ever be retrieved, and no error reports it.
+
+A **chunking strategy** decides how a document becomes vectors. Set it with `CHUNK_STRATEGY` on a model-backed column:
+
+| Strategy | Vectors per document | What it does |
+|---|---|---|
+| `truncate` | 1 | Embeds as much as fits the model's window and drops the rest. The default, and the historical behavior. |
+| `mean` | 1 | Splits the whole document, embeds every piece, and averages them into one vector. No tail loss, but a document covering several topics collapses to their average. |
+| `fixed` | N | Fixed-size windows of `MAX_TOKENS` tokens. |
+| `recursive` | N | Splits on a separator hierarchy: paragraph, then line, then sentence, then space; keeping each piece within `MAX_TOKENS`. |
+| `sentence` | N | Sentence boundaries, packed up to `MAX_TOKENS`. |
+
+`truncate` and `mean` produce one vector per document and work on a [`float_vector`](../Creating_a_table/Data_types.md#Float-vector) column. `fixed`, `recursive` and `sentence` produce several, so they require a [`float_vector_array`](../Creating_a_table/Data_types.md#Float-vector-array) column; using one on a plain `float_vector` is rejected.
+
+The difference is what a match means. With one vector per document, search asks "is this document, as a whole, similar to the query?", and a single relevant paragraph is diluted by everything around it. With one vector per chunk, it asks "does this document *contain* something similar?": each chunk competes on its own, and the document is returned once, scored by its closest chunk (see [Multiple vectors per document](../Searching/KNN.md#Multiple-vectors-per-document)).
+
+**Options**, all valid only alongside `MODEL_NAME` and `KNN_TYPE='hnsw'`:
+
+* `CHUNK_STRATEGY`: one of the five above. Default `truncate`.
+* `MAX_TOKENS`: chunk size in tokens. `0` (default) means the model's own limit; a larger value is clamped down to it.
+* `OVERLAP_TOKENS`: how many tokens consecutive chunks share, so an idea split across a boundary still appears whole in one of them. Requires an explicit non-zero `MAX_TOKENS`. A large overlap is reduced so that chunks still advance through the document — currently anything above half of `MAX_TOKENS` is capped at half.
+* `MAX_CHUNKS`: ceiling on vectors per document. `0` (default) means unlimited.
+
+Important points:
+
+* **`MAX_CHUNKS` discards text.** On overflow the remainder is merged into the last kept chunk, which then exceeds `MAX_TOKENS` and is truncated when embedded. Nothing is left as a visible gap, but the tail is gone.
+* **Local and remote models chunk differently.** Local models split on the model's real tokens. Remote API models (OpenAI, Voyage, Jina) have no local tokenizer and use a conservative byte estimate instead, so the same text and settings will produce a different number of chunks than a local model would.
+
+`ALTER TABLE ... ADD COLUMN` with a model-backed `float_vector_array`, and `ALTER TABLE ... REBUILD EMBEDDINGS` on one, are not supported yet; existing rows can not be backfilled, so the column would stay empty. Declare such a column when creating the table. Both work normally for a `float_vector` column, including with `mean`.
+
+<!-- example chunking -->
+
+<!-- intro -->
+##### SQL:
+
+<!-- request SQL -->
+
+```sql
+-- one vector per sentence group, filled automatically from the text
+CREATE TABLE articles (
+  title text,
+  content text,
+  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
+     model_name='Xenova/all-MiniLM-L6-v2' from='title,content'
+     chunk_strategy='sentence' max_tokens='256' overlap_tokens='32'
+);
+
+INSERT INTO articles (id, title, content) VALUES (1, 'Rotating certificates', 'A long guide with many sections ...');
+
+SELECT id, knn_dist() FROM articles WHERE knn(chunks, 5, 'how do I rotate a certificate');
+```
+
+<!-- intro -->
+##### JSON:
+
+<!-- request JSON -->
+
+```JSON
+POST /cli -d "CREATE TABLE articles (title text, content text, chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine' model_name='Xenova/all-MiniLM-L6-v2' from='title,content' chunk_strategy='sentence' max_tokens='256')"
+
+POST /search
+{
+  "table": "articles",
+  "knn": { "field": "chunks", "query": "how do I rotate a certificate", "k": 5 }
+}
+```
+
+<!-- end -->
+
 ### Vector quantization
 
 HNSW indexes need to be fully loaded into memory to perform KNN search, which can lead to significant memory consumption. To reduce memory usage, scalar quantization can be applied - a technique that compresses high-dimensional vectors by representing each component (dimension) with a limited number of discrete values. Manticore supports 8-bit and 1-bit quantization, meaning each vector component is compressed from a 32-bit float to 8 bits or even 1 bit, reducing memory usage by 4x or 32x, respectively. These compressed representations also allow for faster distance calculations, as more vector components can be processed in a single SIMD instruction. Although scalar quantization introduces some approximation error, it is often a worthwhile trade-off between search accuracy and resource efficiency. For even better accuracy, quantization can be combined with rescoring and oversampling: more candidates are retrieved than requested, and distances for these candidates are recalculated using the original 32-bit float vectors.