fix(plugins): byte-cap and well-form text parts in claude-code capture paths - #4135
Open
michaeltarleton wants to merge 1 commit into
Open
Conversation
1 task
There was a problem hiding this comment.
Pull request overview
This PR hardens the Claude Code memory plugin’s capture flows by byte-capping outgoing text parts and ensuring they are well‑formed UTF‑16, preventing oversized or surrogate-broken transcripts from poisoning the server’s vectordb write pipeline.
Changes:
- Add a shared
capTextParts/capTextPartBytesutility to cap text parts by UTF‑8 bytes and repair lone surrogates. - Apply the cap/sanitization in
auto-capture,subagent-stop, anddebug-capturesend paths. - Add
node:testunit coverage for byte caps and surrogate well‑formedness.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| examples/claude-code-memory-plugin/scripts/lib/text-part-cap.mjs | New shared text-part byte cap + UTF‑16 well‑formedness sanitizer used by capture paths. |
| examples/claude-code-memory-plugin/scripts/lib/text-part-cap.test.mjs | Unit tests validating cap behavior (size + surrogate repair). |
| examples/claude-code-memory-plugin/scripts/auto-capture.mjs | Routes structured parts through capTextParts before sending. |
| examples/claude-code-memory-plugin/scripts/subagent-stop.mjs | Caps structured parts for subagent turn forwarding. |
| examples/claude-code-memory-plugin/scripts/debug-capture.mjs | Caps the debug “addMessage” content to avoid poisoning the debug target server. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+41
to
+69
| export function capTextPartBytes(t, maxBytes = TEXT_PART_MAX_BYTES) { | ||
| const wellFormed = toWellFormedText(t); | ||
| if (Buffer.byteLength(wellFormed, "utf8") <= maxBytes) return wellFormed; | ||
| const budget = Math.max(0, maxBytes - MARKER_RESERVE_BYTES); | ||
| let s = wellFormed.slice(0, budget); | ||
| while (Buffer.byteLength(s, "utf8") > budget) { | ||
| s = s.slice(0, Math.floor(s.length * 0.9)); | ||
| } | ||
| // slice() cuts by UTF-16 code units and can split a surrogate pair even in | ||
| // well-formed input; re-sanitize so the truncated string stays encodable. | ||
| s = toWellFormedText(s); | ||
| return s + `\n... [truncated, ${t.length - s.length} more chars]`; | ||
| } | ||
|
|
||
| // Cap every text part of a message, enforcing both the per-part cap and the | ||
| // per-message total. Non-text parts pass through untouched. This is the one | ||
| // policy point both capture hooks share -- keep call sites thin so the | ||
| // policy cannot drift between them. | ||
| export function capTextParts(parts, totalBudget = TEXT_TOTAL_MAX_BYTES) { | ||
| let remaining = totalBudget; | ||
| return parts.map((p) => { | ||
| if (p.type !== "text" || typeof p.text !== "string") return p; | ||
| const capped = capTextPartBytes( | ||
| p.text, | ||
| Math.max(0, Math.min(TEXT_PART_MAX_BYTES, remaining)), | ||
| ); | ||
| remaining -= Buffer.byteLength(capped, "utf8"); | ||
| return capped === p.text ? p : { ...p, text: capped }; | ||
| }); |
Comment on lines
+61
to
+65
| test("tiny maxBytes terminates and degrades to marker-only output", () => { | ||
| const capped = capTextPartBytes("x".repeat(100000), 10); | ||
| assert.ok(Buffer.byteLength(capped, "utf8") < 100); | ||
| assert.match(capped, MARKER_RE); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The claude-code memory plugin ships text parts with no size bound on any of its capture paths:
auto-capture.mjssanitizePartsForSendsends user/assistant text parts whole. Claude Code expands slash-command/skill prompt files into the transcript as user turns, so a single turn can carry the full text of a large prompt file. Observed in the wild: a 248KBaddMessagepayload from one skill invocation.subagent-stop.mjspushTurnshas the same gap, and a subagent's first user turn is the parent's Task prompt -- routinely large for the same reasons.debug-capture.mjsPOSTs the transcript's combined text uncapped -- so debugging the incident above would poison the very server being debugged.Tool output is already bounded (server-side externalization plus the pathological-payload cap); plain text parts were not.
What happens server-side
The local vectordb's
bytes_rowformat caps string fields at 65535 UTF-8 bytes (#2967). A record derived from an oversized message fails with:The queue consumer re-enqueues the failed task indefinitely (observed retry cadence ~40 minutes over two days). In our case the retry loop eventually starved the event loop until the HTTP listener stopped answering entirely -- process alive, port unbound, health checks dead -- requiring a manual kill. Same failure mode as #2967, reached via
addMessageinstead of resource upload. #3593 (open) truncates thefieldsJSON blob server-side; this client-side cap complements it as defense in depth and protects servers running any released version today.There is a second, size-independent member of the same poison class: JSON transcripts can legally carry a lone surrogate escape (
\ud83d), and the server's Python utf-8 encode raisessurrogates not allowedon it -- another record that can never serialize.Fix
New
scripts/lib/text-part-cap.mjs(placed in the plugin's ownlib/, not the generatedshared/), applied by all three paths above:capTextParts(parts)-- the single policy point both capture hooks call: caps every text part and passes non-text parts through, enforcingfieldsblob still fit one 65535-byte record.toWellFormed(), with a regex fallback for older Node), and truncation re-sanitizes after slicing sinceslice()cuts by UTF-16 code units and can split a surrogate pair.\n... [truncated, N more chars]shape the other truncation helpers use; tiny byte budgets degrade to marker-only output rather than looping.Testing
scripts/lib/text-part-cap.test.mjs(node:test), 8/8 pass: in-bounds passthrough; in-bounds lone-surrogate repair; oversized ASCII and multi-byte inputs capped under the limit with marker; a surrogate-alignment sweep proving no unpaired surrogate survives truncation at any pair boundary (checked via UTF-8 round-trip equality -- the property the server's encode requires); tiny-budget termination; per-message total budget across many parts; untouched pass-through for in-bounds messages.node --checkpasses on all modified scripts.auto-capture.test.mjssuite fails identically on unmodifiedmainin this Windows environment (temp-path ENOENT), so it could not be used as a regression signal here; the new unit tests cover the added behavior directly.Known remaining gaps (intentionally out of scope)
tool_inputparts are sent verbatim and, unliketool_output, are not externalized server-side -- a very large tool input (e.g. a Task prompt inside atool_useblock, or a Write carrying a large file body) can still overflow the derived record. Left out because truncating structured tool input safely needs a server-contract decision; happy to follow up.fieldsblob) plus a retry limit / dead-letter for records that fail serialization non-transiently -- without one, any producer that bypasses this plugin (SDKs, other harness plugins, bots) can still wedge the queue consumer.