correctness.md

Storage, sync, and deletion

Storage

Core storage includes sources, documents, content_units, entities, document_entities, chunks, sync_runs, source_checkpoints, schema_migrations, and one active row in embedding_profiles. chunks.embedding is the authoritative vector and chunks.embedding_status is the authoritative queue state.

Two views and three functions carry the read path: document_context backs rag.documents.list and .get, agent_schema_catalog backs rag.schema.describe, and search_chunks_keyword, search_chunks_semantic, and search_chunks_hybrid back the search tools.

Sync correctness

The partial unique index on running sync_runs rows prevents concurrent syncs for the same source. create_sync_run reports a clear busy error. A run older than 25 hours may be marked failed automatically; this is longer than the 24-hour Workflow maximum. That recovery is lazy rather than scheduled — it runs inside create_sync_run for the same source — so a source that is never dispatched again keeps its abandoned run.

There is no script or flag to recover one explicitly. Operators do it with SQL:

UPDATE sync_runs SET status = 'failed', error = 'operator recovery',
       finished_at = now()
 WHERE source_id = 'json' AND status = 'running';

No renewable source lease exists.

Provider requests use configured batches. Errors are classified as transient, fatal, or unmistakable input errors. Only unmistakable input errors are recursively bisected; a failing singleton is dead-lettered while healthy siblings continue. Transient and fatal batch failures remain pending and fail the task.

The orchestrator drains all pending chunks for its source before checkpointing. Any dead letter, pending row, invalid vector, or profile/fingerprint mismatch fails finalization, leaves the checkpoint unchanged, and marks the production sync run failed. A bounded drain that stops early also fails. Unchanged documents whose persisted chunking fingerprint is stale are re-chunked on replay.

"All pending chunks" has a ceiling: one drain handles at most max_pages × embedding.batch_size chunks, which is 1,000 × 32 = 32,000 by default. Beyond that the run fails without checkpointing and must be drained with scripts/embed.py first. Note also that page size is min(--page-limit, embedding.batch_size), so raising --page-limit above 32 has no effect — increasing throughput means raising embedding.batch_size in rag-engine.yaml, which changes the profile fingerprint and therefore requires the model-change maintenance command.

After repairing bad input, explicitly requeue a bounded set:

python scripts/embed.py --source json --requeue-dead-letters \
  --requeue-limit 100 --max-attempts 3 --reason "source text repaired"

To apply a future changed chunking policy:

python scripts/rechunk.py --source json --limit 100
# Repeat until remaining is zero, then:
python scripts/embed.py --source json

Run these operator commands while normal syncs are paused.

Deletion

JSON records with "deleted": true become sticky logical tombstones in the canonical documents table. The row retains only its source/external identity, kind = 'tombstone' as a queryable marker, deletion timing, status, and a deterministic minimal hash. Document content columns are scrubbed, canonical units/entity links/chunks and vectors are removed, and projection persistence is skipped. Tombstoned references are suppressed before fetch and checked again during persistence, so a later normal payload cannot recreate the record.

Entities referenced only by the deleted document are removed. Shared entities retain their identity, name, and surviving document links, but their mutable metadata is reset to {} so deleted-record details cannot leak through another document.

This template does not infer deletions from records omitted by a source. Gong therefore has no deletion behavior. Any adapter with projection tables must implement and test its own explicit projection cleanup before claiming deletion support; there is no generic projection-deletion framework.

Restoration is a deliberate database administration action performed while syncs are paused. Delete the specific scrubbed tombstone row, then replay the source record:

DELETE FROM documents
WHERE source_id = 'json'
  AND external_id = 'record-id'
  AND status = 'deleted';

Live-database tombstones prevent reimport. Erasure from backups and point-in-time recovery history is an operator retention concern outside this representative template.