Skip to content

fix(langsmith): mount /data for the SmithDB query disk cache - #920

Open
ericdong-langchain wants to merge 1 commit into
mainfrom
fix/smithdb-query-data-volume
Open

fix(langsmith): mount /data for the SmithDB query disk cache#920
ericdong-langchain wants to merge 1 commit into
mainfrom
fix/smithdb-query-data-volume

Conversation

@ericdong-langchain

Copy link
Copy Markdown
Contributor

Summary

The SmithDB query deployment enables the Vortex disk cache but never gets a writable /data volume, so the cache silently never initialises.

templates/smithdb/query-deployment.yaml sets SMITHDB_QUERY__VORTEX_CACHE__DISK__LIMIT from limits.ephemeral-storage whenever that limit exists (it does by default, 200Gi). In SmithDB, setting disk.limit is what turns disk caching on and derives /data/segment and /data/filter from the default disk.path = "/data". But values.yaml shipped volumes: [] / volumeMounts: [] for query, so the container tries to create those directories at the image root as a non-root user and gets EACCES.

This fails soft. The cache coordinator installs a memory-only cache first and the gRPC health server reports Serving before the disk handoff even starts, so the pod stays Ready and just serves every cache miss from object storage. The only signal is a throttled log line every 15s:

ERROR smithdb_query::cache_coordinator: degraded cache: serving memory-only while
  disk cache handoff is incomplete
  lock_paths=["/data/filter/.cache.lock", "/data/segment/.cache.lock"]
  error=PermissionDenied (os error 13)

A self-hosted user hit this in production and ran on a fully cold read path for days after their cutover without noticing.

ingestion and compactionWorker already ship an emptyDir at /data; query was just missed. Our own GCP prod values mount it too, with a comment noting it is required for the /data disk caches — so this brings the public chart in line with how we actually run query.

Changes

  • values.yaml: give smithdb.query.deployment an emptyDir local-ssd-storage mounted at /data, sizeLimit matching the 200Gi ephemeral-storage limit, mirroring ingestion.
  • Unit test asserting the query pod renders the volume and mount.
  • Regenerated README.md via helm-docs; bumped chart to 0.17.0-rc.4.

Test plan

  • helm unittest charts/langsmith -f 'tests/langsmith_smithdb_test.yaml' — 39 passed, including the new case
  • helm lint charts/langsmith clean
  • Full suite shows the same 2 pre-existing JuiceFS annotation failures before and after this change (local helm-unittest 1.1.0 vs CI 0.7.0), unrelated
  • Confirm on a real install that /data/segment and /data/filter initialise and the degraded-cache log stops

Follow-ups (not in this PR)

  • DISK__LIMIT tracks 100% of the ephemeral-storage limit and the cache fills to disk.usable_percent (95%) of it. Combined with emptyDir usage counting against the same limit, this leaves little headroom for logs and scratch. Worth a smaller default.
  • There is no metric or readiness signal for the degraded-cache state — it exists only as a log line, which is why this went unnoticed for days.
  • Needs a backport to v16-stable; self-hosted users on 0.16.x are the ones affected today.

Made with Cursor

The query deployment sets SMITHDB_QUERY__VORTEX_CACHE__DISK__LIMIT from its
ephemeral-storage limit, which enables the Vortex disk cache at /data, but the
chart shipped no volume there. The cache fails to acquire its lock files, logs
"degraded cache: serving memory-only", and serves every miss from object
storage while the pod stays Ready.

Mount an emptyDir at /data to match ingestion, compactionWorker, and how query
already runs in our own deployments.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ericdong-langchain
ericdong-langchain requested a review from a team as a code owner August 6, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant