Skip to content

docs(destination-chroma): correct sync modes and document all config options #84385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
devin-ai-integration wants to merge 3 commits into
base: master
Choose a base branch
from
+81 −54
Draft

docs(destination-chroma): correct sync modes and document all config options #84385

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7ef7840
docs(destination-chroma): correct sync modes, document all config opt…
devin-ai-integration[bot] Aug 13, 2026
26157f9
docs(destination-chroma): soften unverified claims
devin-ai-integration[bot] Aug 13, 2026
06b6ace
docs(destination-chroma): normalize changelog table
devin-ai-integration[bot] Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 81 additions & 54 deletions docs/integrations/destinations/chroma.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,88 +1,115 @@
# Chroma

This page guides you through the process of setting up the [Chroma](https://docs.trychroma.com/?lang=py) destination connector.
This page guides you through setting up the [Chroma](https://docs.trychroma.com/) destination connector.

#### Output Schema
The connector splits the text fields of your records into chunks, turns each chunk into an embedding vector, and writes the vectors, the chunk text, and the record metadata to a single Chroma collection. You can either let Airbyte compute the embeddings with an external service, pass embeddings that already exist in your records, or let Chroma compute them with its own default embedding function.

Only one stream will exist to collect data from all source streams. This will be in a [collection](https://docs.trychroma.com/usage-guide#using-collections) in [Chroma](https://docs.trychroma.com/?lang=py) whose name will be defined by the user, and validated and corrected by Airbyte.
## Prerequisites

For each record, a UUID string is generated and used as the document id. The embeddings generated as defined will be stored as embeddings. Data in the text fields will be stored as documents and those in the metadata fields will be stored as metadata.
- A Chroma database. The connector can connect to a Chroma server over HTTP, or read and write a database directory on the machine running Airbyte.
- A username and password, if your Chroma server requires basic authentication.
- An API key for OpenAI, Azure OpenAI, Cohere, or another OpenAI-compatible embedding service, if you want Airbyte to compute the embeddings. You don't need one if your records already contain embeddings or if you use Chroma's default embedding function.

## Supported sync modes

| Sync mode | Supported? |
| :--- | :--- |
| [Full Refresh - Overwrite](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/full-refresh-overwrite) | Yes |
| [Full Refresh - Append](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/full-refresh-append) | Yes |
| [Full Refresh - Overwrite + Deduped](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/full-refresh-overwrite-deduped) | Yes |
| [Incremental Sync - Append](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append) | Yes |
| [Incremental Sync - Append + Deduped](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped) | Yes |

## Getting Started \(Airbyte Open Source\)
Overwrite syncs don't delete the collection. At the start of the sync, the connector deletes the documents whose `_ab_stream` metadata value matches the streams being overwritten, so documents from other streams and from other tools stay in place.

You can connect to a Chroma instance either in client/server mode or in a local persistent mode. For the local persistent mode, the database file will be saved in the path defined in the `path` config parameter. Note that `path` must be an absolute path, prefixed with `/local`.
Deduplicating syncs require a primary key on the stream. Before writing a record, the connector deletes the documents whose `_ab_record_id` metadata value matches that record's primary key.

:::danger
## Output schema

Persistent Client mode is not supported on Kubernetes
All source streams are written to one Chroma collection, which you name in the connector configuration. Each chunk becomes one document in that collection:

:::
- The document ID is a randomly generated UUID. Source primary keys are stored in metadata instead, so you can't look a record up by its source ID.
- The chunk text is stored as the document, unless you enable **Do not store raw text**.
- The embedding is stored as the document's embedding. If you use Chroma's default embedding function, Airbyte sends no embedding and Chroma computes one.
- The fields you select as metadata are stored as document metadata, along with `_ab_stream` (the stream identifier, in the form `namespace_stream` when the stream has a namespace) and `_ab_record_id` (the record's primary key, for deduplicating streams).

Chroma metadata values must be strings, numbers, or booleans. The connector JSON-encodes any other value, such as an object or an array, into a string.

## Namespace support

Source namespaces don't create separate Chroma collections. Everything is written to the collection you configure, and the namespace becomes part of the `_ab_stream` metadata value, which you can filter on when you query Chroma.

## Set up the Chroma destination

### Connection mode

By default, the `LOCAL_ROOT` env variable in the `.env` file is set `/tmp/airbyte_local`.
Choose one of the following connection modes.

The local mount is mounted by Docker onto `LOCAL_ROOT`. This means the `/local` is substituted by `/tmp/airbyte_local` by default.
**Client/Server Mode** connects to a running Chroma server:

:::caution
- **Host**: the hostname of the Chroma instance, for example `localhost`.
- **Port**: the port the Chroma instance listens on, for example `8000`.
- **SSL**: whether to connect over HTTPS.
- **Username** and **Password**: only needed if the server is configured for basic authentication. Leave both empty otherwise.

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.
Make sure the Chroma server is reachable from Airbyte. If it runs inside a VPC, allow access from the IP address that Airbyte syncs from.

**Persistent Client Mode** stores the database in a directory on the machine running Airbyte:

- **Path**: an absolute path prefixed with `/local`, for example `/local/chroma`.

In Docker deployments, `/local` is mapped to the directory Airbyte mounts for local files, which is `/tmp/airbyte_local` unless you change the `LOCAL_ROOT` environment variable. On macOS, Docker Desktop must be allowed to share `/tmp` and `/private`, because `/tmp` is a symlink to `/private/tmp`. Grant access in **Settings** > **Resources** > **File sharing**, then apply and restart.

:::danger

Persistent Client Mode isn't supported on Kubernetes deployments. Use Client/Server Mode instead.

:::

#### Requirements
### Collection name

To use the Chroma destination, you'll need:
Set **Collection Name** to the collection you want to load data into. The connector creates the collection if it doesn't exist. Chroma uses the name in URLs, so it must:

- An account with API access for OpenAI, Cohere (depending on which embedding method you want to use) or neither (if you want to use the [default chroma embedding function](https://docs.trychroma.com/embeddings#default-all-minilm-l6-v2))
- A Chroma db instance (client/server mode or persistent mode)
- Credentials (for cient/server mode)
- Local File path (for Persistent mode)
- Be between 3 and 63 characters long.
- Start and end with a lowercase letter or a digit.
- Contain only alphanumeric characters, dots, dashes, and underscores.
- Contain no two consecutive dots.
- Not be a valid IPv4 address.

#### Configure Network Access
The connector checks these rules during the connection test and reports a specific error when the name is invalid.

Make sure your Chroma database can be accessed by Airbyte. If your database is within a VPC, you may need to allow access from the IP you're using to expose Airbyte.
### Embedding

### Setup the Chroma Destination in Airbyte
Pick how the embeddings are produced:

You should now have all the requirements needed to configure Chroma as a destination in the UI. You'll need the following information to configure the Chroma destination:
- **OpenAI**: uses `text-embedding-ada-002` with 1536 dimensions. Requires an **OpenAI API key**.
- **Azure OpenAI**: uses `text-embedding-ada-002` with 1536 dimensions from your own Azure resource. Requires the **Azure OpenAI API key**, **Resource base URL**, and **Deployment**.
- **Cohere**: requires a **Cohere API key**.
- **OpenAI-compatible**: for self-hosted or third-party services that expose the OpenAI embeddings API. Requires the **Base URL**, **Embedding dimensions**, and, depending on the service, an **API key** and **Model name**.
- **From Field**: uses an embedding that already exists in the record. Set **Field name** to the field holding the vector and **Embedding dimensions** to its length. Records whose vector has a different length fail the sync.
- **Fake**: random vectors with 1536 dimensions, for testing a pipeline without paying for embeddings.
- **Chroma Default Embedding Function**: Airbyte sends no embedding, and Chroma embeds the documents itself with its [default embedding function](https://docs.trychroma.com/docs/embeddings/embedding-functions), which runs the `all-MiniLM-L6-v2` model on the machine running Chroma. No API key is needed.

- (Required) **Text fields to embed**
- (Optional) **Text splitter** Options around configuring the chunking process provided by the [Langchain Python library](https://python.langchain.com/docs/get_started/introduction).
- (Required) **Fields to store as metadata**
- (Required) **Collection** The name of the collection in Chroma db to store your data
- (Required) Authentication method
- For client/server mode
- **Host** for example localhost
- **Port** for example 8000
- **Username** (Optional)
- **Password** (Optional)
- For persistent mode
- **Path** The path to the local database file. Note that `path` must be an absolute path, prefixed with `/local`.
- (Optional) Embedding
- **OpenAI API key** if using OpenAI for embedding
- **Cohere API key** if using Cohere for embedding
- Embedding **Field name** and **Embedding dimensions** if getting the embeddings from stream records
### Processing

## Namespace support
These options control how records become chunks:

- **Chunk size** (required): the maximum size of a chunk in tokens, up to 8191. Keep it small enough for the context window of the model you query with.
- **Chunk overlap**: how many tokens consecutive chunks share, which helps preserve context across chunk boundaries. Defaults to 0.
- **Text fields to embed**: the record fields to embed. Use dot notation for nested fields, such as `user.name`, and wildcards for arrays, such as `users.*.name`. If you leave this empty, all fields are embedded.
- **Fields to store as metadata**: the record fields to store as document metadata. If you leave this empty, all fields are stored as metadata.
- **Text splitter**: how to split text that exceeds the chunk size. You can split by separator, by Markdown header level, or by the syntax of a programming language.
- **Field name mappings**: renames source fields before they're written.
- **Do not store raw text**: stores only the embedding and metadata. Use this when the source text is sensitive or already available elsewhere. Retrieval-augmented generation workflows that read the chunk text from Chroma break when you enable it.

This destination supports [namespaces](https://docs.airbyte.com/platform/using-airbyte/core-concepts/namespaces).
The connector writes documents to Chroma in batches of 128 chunks.

## Changelog

<details>
<summary>Expand to review</summary>

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :-------------------------------------------------------- |:-------------------------------------------------------------|
| Version | Date | Pull Request | Subject |
| :------ | :--- | :----------- | :------ |
| 0.0.55 | 2026-08-13 | [84361](https://github.com/airbytehq/airbyte/pull/84361) | Update the CDK to remediate CVE-2025-68664 in the langchain dependency |
| 0.0.54 | 2025-05-03 | [59326](https://github.com/airbytehq/airbyte/pull/59326) | Update dependencies |
| 0.0.53 | 2025-04-26 | [58256](https://github.com/airbytehq/airbyte/pull/58256) | Update dependencies |
Expand Down Expand Up @@ -128,15 +155,15 @@ This destination supports [namespaces](https://docs.airbyte.com/platform/using-a
| 0.0.13 | 2024-06-25 | [40431](https://github.com/airbytehq/airbyte/pull/40431) | Update dependencies |
| 0.0.12 | 2024-06-23 | [40222](https://github.com/airbytehq/airbyte/pull/40222) | Update dependencies |
| 0.0.11 | 2024-06-22 | [40068](https://github.com/airbytehq/airbyte/pull/40068) | Update dependencies |
| 0.0.10 | 2024-04-15 | [#37333](https://github.com/airbytehq/airbyte/pull/37333) | Updated CDK & pytest version to fix security vulnerabilities |
| 0.0.9 | 2023-12-11 | [#33303](https://github.com/airbytehq/airbyte/pull/33303) | Fix bug with embedding special tokens |
| 0.0.8 | 2023-12-01 | [#32697](https://github.com/airbytehq/airbyte/pull/32697) | Allow omitting raw text |
| 0.0.7 | 2023-11-16 | [#32608](https://github.com/airbytehq/airbyte/pull/32608) | Support deleting records for CDC sources |
| 0.0.6 | 2023-11-13 | [#32357](https://github.com/airbytehq/airbyte/pull/32357) | Improve spec schema |
| 0.0.5 | 2023-10-23 | [#31563](https://github.com/airbytehq/airbyte/pull/31563) | Add field mapping option |
| 0.0.4 | 2023-10-15 | [#31329](https://github.com/airbytehq/airbyte/pull/31329) | Add OpenAI-compatible embedder option |
| 0.0.3 | 2023-10-04 | [#31075](https://github.com/airbytehq/airbyte/pull/31075) | Fix OpenAI embedder batch size |
| 0.0.2 | 2023-09-29 | [#30820](https://github.com/airbytehq/airbyte/pull/30820) | Update CDK |
| 0.0.1 | 2023-09-08 | [#30023](https://github.com/airbytehq/airbyte/pull/30023) | 🎉 New Destination: Chroma (Vector Database) |
| 0.0.10 | 2024-04-15 | [#37333](https://github.com/airbytehq/airbyte/pull/37333) | Updated CDK & pytest version to fix security vulnerabilities |
| 0.0.9 | 2023-12-11 | [#33303](https://github.com/airbytehq/airbyte/pull/33303) | Fix bug with embedding special tokens |
| 0.0.8 | 2023-12-01 | [#32697](https://github.com/airbytehq/airbyte/pull/32697) | Allow omitting raw text |
| 0.0.7 | 2023-11-16 | [#32608](https://github.com/airbytehq/airbyte/pull/32608) | Support deleting records for CDC sources |
| 0.0.6 | 2023-11-13 | [#32357](https://github.com/airbytehq/airbyte/pull/32357) | Improve spec schema |
| 0.0.5 | 2023-10-23 | [#31563](https://github.com/airbytehq/airbyte/pull/31563) | Add field mapping option |
| 0.0.4 | 2023-10-15 | [#31329](https://github.com/airbytehq/airbyte/pull/31329) | Add OpenAI-compatible embedder option |
| 0.0.3 | 2023-10-04 | [#31075](https://github.com/airbytehq/airbyte/pull/31075) | Fix OpenAI embedder batch size |
| 0.0.2 | 2023-09-29 | [#30820](https://github.com/airbytehq/airbyte/pull/30820) | Update CDK |
| 0.0.1 | 2023-09-08 | [#30023](https://github.com/airbytehq/airbyte/pull/30023) | 🎉 New Destination: Chroma (Vector Database) |

</details>
Loading