fix(langsmith): mount /data for the SmithDB query disk cache - #920
Open
ericdong-langchain wants to merge 1 commit into
Open
fix(langsmith): mount /data for the SmithDB query disk cache#920ericdong-langchain wants to merge 1 commit into
ericdong-langchain wants to merge 1 commit into
Conversation
The query deployment sets SMITHDB_QUERY__VORTEX_CACHE__DISK__LIMIT from its ephemeral-storage limit, which enables the Vortex disk cache at /data, but the chart shipped no volume there. The cache fails to acquire its lock files, logs "degraded cache: serving memory-only", and serves every miss from object storage while the pod stays Ready. Mount an emptyDir at /data to match ingestion, compactionWorker, and how query already runs in our own deployments. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The SmithDB
querydeployment enables the Vortex disk cache but never gets a writable/datavolume, so the cache silently never initialises.templates/smithdb/query-deployment.yamlsetsSMITHDB_QUERY__VORTEX_CACHE__DISK__LIMITfromlimits.ephemeral-storagewhenever that limit exists (it does by default, 200Gi). In SmithDB, settingdisk.limitis what turns disk caching on and derives/data/segmentand/data/filterfrom the defaultdisk.path = "/data". Butvalues.yamlshippedvolumes: []/volumeMounts: []for query, so the container tries to create those directories at the image root as a non-root user and getsEACCES.This fails soft. The cache coordinator installs a memory-only cache first and the gRPC health server reports
Servingbefore the disk handoff even starts, so the pod stays Ready and just serves every cache miss from object storage. The only signal is a throttled log line every 15s:A self-hosted user hit this in production and ran on a fully cold read path for days after their cutover without noticing.
ingestionandcompactionWorkeralready ship an emptyDir at/data;querywas just missed. Our own GCP prod values mount it too, with a comment noting it is required for the/datadisk caches — so this brings the public chart in line with how we actually run query.Changes
values.yaml: givesmithdb.query.deploymentan emptyDirlocal-ssd-storagemounted at/data,sizeLimitmatching the 200Gi ephemeral-storage limit, mirroringingestion.README.mdvia helm-docs; bumped chart to0.17.0-rc.4.Test plan
helm unittest charts/langsmith -f 'tests/langsmith_smithdb_test.yaml'— 39 passed, including the new casehelm lint charts/langsmithclean/data/segmentand/data/filterinitialise and the degraded-cache log stopsFollow-ups (not in this PR)
DISK__LIMITtracks 100% of the ephemeral-storage limit and the cache fills todisk.usable_percent(95%) of it. Combined with emptyDir usage counting against the same limit, this leaves little headroom for logs and scratch. Worth a smaller default.v16-stable; self-hosted users on 0.16.x are the ones affected today.Made with Cursor