feat: share and compress Lumen indexes across worktrees

.gitattributes

@@ -1,6 +1,10 @@
 bench-results linguist-generated
 docs/plans linguist-generated
 
+# Keep the vendored sqlite-vec release source byte-for-byte identical to
+# upstream, including its existing whitespace.
+internal/sqlitevec/sqlite-vec.[ch] -text -whitespace linguist-vendored
+
 # Windows batch files require CRLF line endings — the cmd.exe parser
 # has 512-byte boundary bugs with bare-LF files (GOTO/CALL label
 # parsing). Pin .bat/.cmd to CRLF regardless of the user's

CLAUDE.md

@@ -130,6 +130,7 @@ Codex, Cursor, and OpenCode reuse the same repo-root `skills/`, `hooks/`, and
 | `OLLAMA_HOST`            | `http://localhost:11434` | Ollama server URL                          |
 | `LM_STUDIO_HOST`         | `http://localhost:1234`  | LM Studio server URL                       |
 | `LUMEN_MAX_CHUNK_TOKENS` | `512`                    | Max tokens per chunk before splitting      |
+| `LUMEN_VECTOR_STORAGE`   | `int8`                   | Vector precision (`int8` or `float32`)     |
 
 ¹ `ordis/jina-embeddings-v2-base-code` (Ollama),
 `nomic-ai/nomic-embed-code-GGUF` (LM Studio)
@@ -157,7 +158,8 @@ Codex, Cursor, and OpenCode reuse the same repo-root `skills/`, `hooks/`, and
 ├── internal/
 │   ├── config/         # Config loading & paths
 │   ├── index/          # Orchestration (Merkle + embedding + chunking)
-│   ├── store/          # SQLite + sqlite-vec operations
+│   ├── store/          # Shared SQLite collection + sqlite-vec operations
+│   ├── sqlitevec/      # Vendored sqlite-vec v0.1.9 wrapper and sources
 │   ├── chunker/        # Go AST parsing → chunks
 │   ├── embedder/       # Ollama/LM Studio HTTP client
 │   └── merkle/         # Change detection (SHA-256 tree)
@@ -189,8 +191,9 @@ because slog writes to the log file while tui writes to the process stderr.
 ## Key Design Decisions
 
 - **Merkle tree for diffs**: Avoid re-indexing unchanged code
-- **Model name + IndexVersion in DB path**: Different models or index versions →
-  separate indexes (SHA-256 hash of path + model name + `IndexVersion`).
+- **Repository collection profile in DB path**: Git common directory, indexed
+  scope, model, dimensions, vector precision, chunking profile, and
+  `IndexVersion` select a shared content-addressed collection.
   `IndexVersion` is a hardcoded constant in `internal/config/version.go` —
   increment it (and document why in the commit message) whenever a chunker,
   embedder, or index-format change would make existing indexes incompatible. Do
@@ -199,13 +202,19 @@ because slog writes to the log file while tui writes to the process stderr.
   .gitattributes → extension
 - **Chunk splitting at line boundaries**: Oversized chunks split at
   `LUMEN_MAX_CHUNK_TOKENS` (512 default)
-- **32-batch embedding**: Balance memory vs. API round-trips
+- **256-batch embedding**: Only exact embedding inputs missing from the shared
+  vector table are sent to the backend
 - **Cosine distance KNN**: Normalized for semantic similarity
+- **Vendored sqlite-vec**: v0.1.9 is compiled behind `internal/sqlitevec` so
+  collection deletion and vector behavior do not drift with system packages
 - **Plugin system**: Declarative Claude and Cursor packaging at the repo root,
   plus Codex/OpenCode install surfaces that reuse the same skills and launcher
 - **No repo-root `.mcp.json`**: Use `mcp.json` for Cursor and `.codex/INSTALL.md`
   for Codex so Claude project behavior never changes implicitly
 
+See [docs/INDEX_STORAGE.md](docs/INDEX_STORAGE.md) for collection identity,
+deduplication, vector precision, migration, cleanup, and status-field semantics.
+
 ## Claude Integration Notes
 
 When planning any work related to claude code plugin, marketplace, hooks,

README.md

@@ -216,8 +216,9 @@ Files → semantic chunks → vector embeddings → SQLite/sqlite-vec → KNN se
 
 When Claude needs to understand code, it calls `semantic_search` instead of
 reading entire files. The index is stored outside your repo
-(`~/.local/share/lumen/<hash>/index.db`), keyed by project path and model name —
-different models never share an index.
+(`~/.local/share/lumen/<hash>/index.db`). Git worktrees from the same repository
+use one collection for a compatible model, vector-storage, and chunking profile;
+non-Git projects use private collections. Different profiles never collide.
 
 ## Benchmarks
 
@@ -284,6 +285,7 @@ All configuration is via environment variables:
 | `OLLAMA_HOST`            | `http://localhost:11434` | Ollama server URL                                             |
 | `LM_STUDIO_HOST`         | `http://localhost:1234`  | LM Studio server URL                                          |
 | `LUMEN_MAX_CHUNK_TOKENS` | `512`                    | Max tokens per chunk before splitting                         |
+| `LUMEN_VECTOR_STORAGE`   | `int8`                   | Vector precision (`int8` or `float32`)                         |
 | `LUMEN_EMBED_DIMS`       | —                        | Override embedding dimensions (required for unlisted models)  |
 | `LUMEN_EMBED_CTX`        | `8192` (unlisted models) | Override context window length                                |
 
@@ -387,10 +389,12 @@ Index databases are stored outside your project:
 ~/.local/share/lumen/<hash>/index.db
 ```
 
-Where `<hash>` is derived from the absolute project path, embedding model name,
-and binary version. Different models or Lumen versions automatically get
-separate indexes. No files are added to your repo, no `.gitignore` modifications
-needed.
+Where `<hash>` identifies the Git common directory (or the absolute path for a
+non-Git project), indexed scope, embedding model and dimensions, vector
+precision, chunking profile, and index version. Worktrees in one repository
+share content-addressed file revisions and vectors while retaining independent
+project memberships. Vectors use int8 storage by default; set
+`LUMEN_VECTOR_STORAGE=float32` to opt out. No files are added to your repo.
 
 You can safely delete the entire `lumen` directory to clear all indexes, or let
 Lumen reclaim the space for you:
@@ -405,11 +409,15 @@ An index counts as used every time Lumen opens it (search, indexing, status, or
 session start), so indexes for projects you still work on are never removed.
 Indexes with an indexer currently running are always kept.
 
-**Git worktrees** are detected automatically. When you create a new worktree
-(`git worktree add` or `claude --worktree`), Lumen finds a sibling worktree's
-existing index and copies it as a seed. The Merkle tree diff then re-indexes
-only the files that actually differ — typically a handful of files instead of
-the entire codebase. No configuration needed; it just works.
+**Git worktrees** are detected automatically. A new worktree attaches unchanged
+path-and-content revisions directly from the repository collection and embeds
+only missing chunk inputs. Removing an old worktree drops its memberships;
+shared revisions and vectors remain until their final reference disappears.
+Legacy per-worktree indexes migrate lazily, reusing unchanged float32 vectors
+without contacting the embedding backend.
+
+For the complete storage key, sharing rules, status metrics, migration process,
+and cleanup lifecycle, see [Index storage and lifecycle](docs/INDEX_STORAGE.md).
 
 ## CLI Reference
 
@@ -471,6 +479,19 @@ to the model, set **Override Domain Type** → **Text Embedding**.
 Set `LUMEN_EMBED_MODEL` to a model from the supported table above. Each model
 gets its own database; the old index is not deleted automatically.
 
+Changing `LUMEN_VECTOR_STORAGE`, `LUMEN_EMBED_DIMS`, or `LUMEN_MAX_CHUNK_TOKENS`
+also selects a separate collection. Run `lumen clean` after the old profile is
+no longer in use if you want to reclaim its disk space.
+
+**Understanding index size and deduplication**
+
+Call `index_status` for the project. It reports project-local file and chunk
+counts alongside collection-wide unique vectors, shared references,
+deduplication ratio, vector precision, database size, and currently reclaimable
+SQLite pages. See
+[Index storage and lifecycle](docs/INDEX_STORAGE.md#reading-index-status) for
+definitions and examples.
+
 **Slow first indexing**
 
 The first run embeds every file. Subsequent runs only process changed files