Skip to content

Latest commit

 

History

History
74 lines (52 loc) · 3.26 KB

spec.md

File metadata and controls

74 lines (52 loc) · 3.26 KB

hexsleuth design

Goal

Let an AI coding agent derive the layout of an unknown binary format from sample files, with evidence for every claim, and emit artifacts that other tools can use.

Workflow

  1. probe each sample.
  2. compare the samples to separate constant and variable regions.
  3. infer candidate fields.
  4. hypothesize: the agent writes a candidate layout.
  5. verify the layout against every sample.
  6. Repeat 3 to 5 until every sample parses.
  7. emit the specification, .ksy, parser and tests.

Tools

Every tool is available through MCP and through the CLI with the same input and output shapes.

Tool Input Output
probe one file, optional byte range size, magic candidates, entropy per block, printable strings with offsets, alignment, repeated patterns
compare two or more files regions that are identical, regions that differ, regions whose length tracks file size
infer_fields files, optional region candidate lengths, counts, offsets, endianness and version fields, each with a confidence score and the samples that support it
detect_compression file, offset detected standard compression and the decompressed size; decompressed bytes are written to a work directory
verify_layout files and a layout in hexsleuth layout JSON per sample: parsed tree, first failing field, unread bytes
read_bytes file, offset, length hex and ASCII view, with optional interpretation as integers or floats
emit verified layout Markdown spec, Kaitai Struct .ksy, TypeScript parser, Vitest tests that parse every sample

Layout JSON

A small description language that maps one to one onto Kaitai Struct:

  • sequences of fields with types u1, u2, u4, u8, signed variants, f4, f8, fixed and length-prefixed byte strings, and strings with an encoding
  • endianness per layout or per field
  • size, repeat and repeat-expr from earlier fields
  • instances at absolute offsets for offset tables
  • nested types

Evidence

Every finding carries:

  • the sample files and byte ranges that support it
  • the check that produced it, for example "value equals remaining length in 5 of 5 samples"
  • a confidence score from 0 to 1
  • known limits, for example "only one sample has a non-empty table"

Encryption and protection

  • High entropy that does not match a supported compression is reported as possibly-encrypted with its range.
  • No tool attempts decryption, key search, XOR guessing or any bypass of protection. This is a hard boundary for the project.

Technical choices

  • TypeScript, Node.js 22 or later, ESM.
  • MCP server through the official TypeScript SDK.
  • Decompression through Node.js built-ins where possible. Add libraries only for LZ4 and Zstandard.
  • Tests with Vitest. Fixtures are generated by scripts in the repository, in open formats or formats designed for the tests.

Showcases

Write each showcase as a step-by-step agent session on an open format:

  1. PNG chunks
  2. ZIP local headers and central directory
  3. WAV RIFF chunks
  4. A sample format designed for this repository, with an offset table, a string pool and a compressed section

Out of scope for the first release

  • Executable code analysis. Tools such as REA already cover it.
  • A graphical hex editor.