Skip to content

fix(export_l5x): repair Region Map truncation and stale-length errors - #48

Merged
hutcheb merged 1 commit into
hutcheb:mainfrom
colinlove:fix/region-map-truncation-and-length-mismatch
Nov 3, 2026
Merged

hutcheb merged 1 commit into
hutcheb:mainfrom
colinlove:fix/region-map-truncation-and-length-mismatch

Conversation

@colinlove

Copy link
Copy Markdown
Contributor

AI-assistance disclosure: This fix was found and written with Claude (Anthropic's AI coding assistant), working from a corpus of ~3,500 real production ACD files, under the author's direction and review. The root-cause analysis and byte-level investigation are genuine; the write-up below is the reviewed product of that work, not unreviewed AI output. Happy to answer questions about the root cause or walk through the fix in more detail.


Region Map records can be truncated or self-report the wrong length

The patch file covers the complete fix — both parts described below.

Symptom

Exporting certain real-world .ACD projects either:

  1. silently drops the last rung of a routine from the exported L5X, with
    no error at all, or
  2. raises ValueError: Invalid Region Map length and aborts the whole
    export, even though the project opens and runs fine in Studio 5000.

Root cause

The "Region Map" comps record is Logix's index from a routine to its rungs.
Each entry is normally a fixed 16 bytes: (object_id, parent_id, unknown, seq). Two independent forms of drift were found in real project files:

  1. A truncated final entry. A routine's last rung can be stored as a
    12-byte (parent_id, unknown, seq) triple with the object_id field
    simply missing, instead of the usual 16 bytes. _parse_region_map
    treated any 12-byte remainder as an inert trailer/footer and silently
    discarded it — so that rung vanished from the export with no warning,
    no error, nothing.
  2. A stale self-described length. The "modern" record format stores its
    own length near the start (at offset 0x4A), which is supposed to equal
    the record's total size. On some real files this stored length is
    smaller than the record's actual on-disk size, by anywhere from a
    handful of bytes to several hundred — the real entries are still present
    past where the stale length says the record ends. The existing code
    treated any mismatch here as fatal (raise ValueError("Invalid Region Map length")), aborting the entire file's export.

Both look like the same underlying pattern: something in Studio 5000
appends to this record without always rewriting its own bookkeeping
(length prefix / object_id) to match — under-recording in the second case,
omitting outright in the first. Both were reproduced against real,
non-synthetic project files (proprietary customer projects — see the
No test files attached note in the top-level README of this
contribution for why no sample .ACD/.L5X is included).

Fix, part 1 — Truncated last entry

_parse_region_map now recognizes a 12-byte trailer whose first field
(parent_id) matches a parent_id already seen among the record's real
entries — a coincidence an inert footer has no reason to produce — and
surfaces it as an entry with object_id=None instead of silently dropping
it. A new _repair_region_map_gaps() (called once the rungs table
exists) recovers the missing object_id by elimination: for the routine
with the gap, every other rung already has a resolved object_id and a
known sequence number; if the gap's sequence number falls strictly between
two already-resolved neighbours, the missing rung must be whichever on-disk
rung (by SQLite rowid, which tracks on-disk insertion order) falls between
them. If more than one candidate qualifies, it logs a warning and leaves
that one rung out rather than guessing wrong.

Fix, part 2 — Stale self-described length (with empirical alignment search)

When the stored 0x4A length doesn't match either the "modern" or "legacy"
expected relationship to the record's actual size, the naive fallback
(assume entries start at fixed offset 0x4E) is often wrong — verified
on real files where the true start was at a different offset, and the
0x4E fallback silently produced garbage entries (worse than the
original error, because it looked successful).

The fix uses an empirical alignment search (_find_region_map_alignment):
brute-force every candidate start byte offset (0 to min(on-disk size − 16,
512)), score each by how many of its would-be 16-byte entries have a
object_id field (the 4th uint32, at entry_start + 12) that matches a
real rung object_id from the rungs table. Only trust an alignment that
matches ≥ 80% of its entries and has ≥ 3 entries total (a clear signal when
it's there — wrong offsets consistently score 0%, correct ones score ≥
98%). The rungs table is populated before populate_region_map runs in
__post_init__, so the known rung ids are always available at this point.

Falls back to the fixed 0x4E offset (with a clearer "no confident
alignment found" warning) when nothing clears that bar, rather than either
raising or silently accepting a plausible-looking wrong answer.

This approach is logged as a log.warning in both the "found a better
alignment" and "falling back to 0x4E" cases — it's a real data-quality
signal worth knowing about per-file.

Suggested validation

Both code paths are logged (log.info for a recovered rung, log.warning
for an unrecoverable gap or a length mismatch) rather than silent, so it's
easy to see on a batch conversion which files hit either path and by how
much. On the fleet of ~770 real project files this was tested against,
roughly 1 in 30 files hit the stale-length path; for those, the empirical
alignment search found the correct offset in all but 2–3 cases (which
received the 0x4E fallback with a warning, rather than garbage data).

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@hutcheb
hutcheb merged commit abd260d into hutcheb:main Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants