fix(vectordb): truncate oversized fields JSON blob before bytes_row write

[TurgutKural]

Description

The local vectordb bytes_row format length-prefixes every string field with a uint16, capping each at 65 535 bytes. The fields column stores all scalar metadata (abstract, description, name, tags, uri, …) as a single JSON blob. When a large abstract (~50 KB after the #2774 cap) is combined with other scalar fields, the total serialized blob can exceed 65 535 bytes, causing:

RuntimeError: string field 'fields' exceeds 65535 bytes

This was partially addressed by #3114 (drop content field for non-VikingDB backends) and #2774 (cap abstract at 50 KB), but the combined JSON blob of all remaining scalars can still overflow.

Root cause

local_collection.py:844:

cands_list[i].fields = safe_json_dumps(data, ensure_ascii=False)

data contains every scalar field from the collection schema (uri, context_type, abstract, description, name, tags, account_id, owner_user_id, created_at, updated_at, level, active_count). The abstract alone can be up to 50 000 bytes after the #2774 truncation. Adding the remaining fields pushes the JSON blob past 65 535 bytes.

Fix

Add LocalCollection._truncate_fields_json() which:

1. Serializes the payload once; if it fits, returns it unchanged (zero overhead for normal records).
2. If oversized, iteratively halves the longest string value until the blob fits.
3. Non-string fields (level, active_count, timestamps) are never modified.
4. As a last resort, drops the offending string field entirely.

The write path now calls self._truncate_fields_json(data) instead of safe_json_dumps(data).

Related

• Fixes the remaining gap after [Bug]: Local vectordb fails on large files due to 65535 byte field limit #2967 / fix(vectordb): only write content field for VikingDB backends #3114
• Related: BUG: Unterminated string JSON parse error in VikingVectorIndexBackend query #2117, fix(memory): truncate memory abstracts before vector writes #2774, [Bug]: Legacy uint16-truncated records are permanently undeletable — delete/upsert still json.loads corrupt old_fields, no repair path for pre-#2171 stores #2966

Type of Change

• Bug fix (non-breaking change that fixes an issue)

Changes Made

• openviking/storage/vectordb/collection/local_collection.py: add _truncate_fields_json() static method + import STRING_MAX_UINT16_LENGTH; replace safe_json_dumps call in _write_data_list
• tests/storage/test_fields_json_truncation.py: 7 unit tests (small passthrough, single large field, multiple large fields, non-string preservation, empty dict, boundary exact fit, multi-byte UTF-8)

Testing

$ python -m pytest tests/storage/test_fields_json_truncation.py -v
7 passed in 0.05s

Verified against a live deployment with 5 memory files (82–131 KB) that previously failed with string field 'fields' exceeds 65535 bytes. After the fix, all 5 records are written successfully.

[michaeltarleton]

Adding a production data point in support of this fix: we hit the exact RuntimeError: string field 'fields' exceeds 65535 bytes failure on 0.4.4 via the claude-code memory plugin's addMessage path (not resource upload). A single ~248KB message part produced a record that could never serialize; the queue consumer re-enqueued it every ~40 minutes for two days, and the retry loop eventually starved the event loop until the HTTP server stopped answering entirely (process alive, port unbound, health checks dead) -- the same server-death mode described in #2967.

Two takeaways from that incident that this PR's approach addresses:

1. The overflow is reachable from ordinary chat-capture traffic, not just large-file resource ingestion, so the truncation needs to live at the fields blob write as this PR does -- upstream callers can't all be trusted to pre-cap.
2. A retry limit / poison-record quarantine for serialization failures would be worth pairing with this: truncation prevents the overflow, but any future non-transient storage error still loops forever today.

I've opened #4135 as a complementary client-side cap for the claude-code plugin's capture paths so oversized (or surrogate-broken) text parts never leave the client, which protects servers on already-released versions -- but the server-side truncation here is the fix that covers every producer.