Enforce canonical encoding for Arkade payloads - #1082
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
WalkthroughThis PR enforces canonical encoding across asset groups and extensions by introducing minimal LEB128 varint decoding, validating serialization integrity, rejecting non-canonical presence bits and opcode encodings, and verifying round-trip consistency through fuzz tests. ChangesCanonical Encoding Validation
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ghost
left a comment
ghost
left a comment
There was a problem hiding this comment.
Arkana Review — #1082
Verdict: Request changes (protocol-critical scope → requires human sign-off regardless)
Overall Assessment
This PR closes encoding-level malleability in the Arkade OP_RETURN payload. The approach is sound: reject at parse time anything that doesn't round-trip to identical bytes. The canonical varint implementation is correct, the fuzz oracles are the right design, and the invariant ("parse ⇒ re-serialize == original") is well-chosen. Good work.
However, I have findings ranging from a real correctness gap to cross-repo coordination concerns.
🔴 P0 — Correctness Gap
1. extension.go: deserializeVarSlice is duplicated — only one copy is patched
pkg/ark-lib/extension/extension.go:283 has its own deserializeVarSlice that is correctly updated to use varint.ReadCanonical. But extension.go also has serializeVarSlice at line ~273 (on master). The concern is the other direction: there is a separate deserializeVarSlice in pkg/ark-lib/asset/utils.go:83 which is also patched. Good — both copies are patched. ✅ Verified on re-read.
However, are there any other call sites of binary.ReadUvarint remaining in pkg/ark-lib/? A grep for binary.ReadUvarint across pkg/ark-lib/ would confirm no stragglers. If any remain, the canonical invariant is broken — a non-canonical varint at that call site passes through undetected.
Action: Confirm zero remaining calls to binary.ReadUvarint in pkg/ark-lib/. If any exist, they must also be migrated to varint.ReadCanonical.
🟡 P1 — Cross-Repo Coordination
2. TypeScript SDK and .NET SDK do NOT validate canonical encoding
Both ts-sdk (src/extension/asset/utils.ts:readVarUint) and dotnet-sdk (BufferReader.cs:ReadVarInt) accept non-canonical LEB128. Neither validates trailing bytes or minimal push opcodes in extension parsing.
This is acceptable today because:
- Both SDKs produce canonical output (their encoders naturally emit minimal LEB128)
- This PR only tightens arkd's acceptance, not what arkd emits
But it creates a consistency gap:
- If a TS/dotnet SDK consumer builds a payload from raw bytes (not via the SDK's own serializer), the SDK will happily accept non-canonical data that arkd will reject
- Future SDK versions should match arkd's strictness to avoid confusion
Action: File tracking issues on arkade-os/ts-sdk and arkade-os/dotnet-sdk to add canonical varint validation + trailing byte rejection. Not a blocker for this PR, but should be tracked.
🟡 P1 — Edge Case in Canonical Validation
3. varint.go:22-28: Reader position is consumed even on rejection
func ReadCanonical(r *bytes.Reader) (uint64, error) {
before := r.Len()
v, err := binary.ReadUvarint(r)
if err != nil {
return 0, err
}
var buf [binary.MaxVarintLen64]byte
if before-r.Len() != binary.PutUvarint(buf[:], v) {
return 0, ErrNonMinimal
}
return v, nil
}When ErrNonMinimal is returned, the reader position has already been advanced past the non-canonical varint bytes. If any caller attempts error recovery (unlikely today, but defensive concern), the reader state is inconsistent.
This is fine for the current callers (they all abort on error), but worth a doc comment: // On error, the reader position is undefined.
Action: Minor — add a doc note that reader position is undefined on error.
🟢 P2 — Suggestions
4. extension.go:220-224: Ordering of trailing-bytes check vs. push-opcode check
The current order is: check trailing bytes → check script error → check minimal push. This is fine functionally, but the script error check (tokenizer.Err()) should arguably come first, since if the tokenizer is in an error state, tokenizer.Next() may return misleading results.
Current code:
if tokenizer.Next() {
return nil, fmt.Errorf("non-canonical extension: trailing data after payload")
}
if err := tokenizer.Err(); err != nil {
return nil, fmt.Errorf("non-canonical extension: invalid script: %w", err)
}btcd's ScriptTokenizer.Next() returns false on error, so tokenizer.Next() returning true is unambiguous. But tokenizer.Next() returning false could mean either "no more ops" or "error" — the subsequent Err() check disambiguates correctly. ✅ Order is actually correct.
5. asset_group.go:239-243: Presence mask is hardcoded — consider deriving from constants
const definedPresenceMask = maskAssetId | maskControlAsset | maskMetadataThis is correct today. If a new optional field is added to AssetGroup and its mask constant is defined but not added here, the decoder will reject valid payloads. The risk is low (any new field requires touching this function anyway), but a compile-time assertion or comment would help.
Action: Minor — add a comment: // IMPORTANT: update this mask when adding new optional fields
6. Tests are thorough. The fuzz oracles (parse(x) ok ⇒ serialize(parse(x)) == x) are exactly the right pattern for canonicality. The targeted rejection tests cover all new checks. The varint test covers zero, boundary (127/128), max-uint64, and overlong variants. Well done.
🔒 Protocol-Critical Flag
This PR modifies deserialization of Arkade payloads — the wire format that encodes asset groups inside VTXOs. While the changes are defense-hardening (reject → not accept), any bug here could cause:
- Valid on-chain payloads to be rejected (consensus split)
- Arkd nodes on different versions to disagree on payload validity
The PR description states: "arkd already emits canonical bytes, so no legitimate flow is affected. This only tightens acceptance of malformed or third-party input. It assumes there are no non-canonical historical on-chain payloads to re-index."
This assumption must be verified. If there are any historical on-chain payloads with non-canonical encoding, this PR will break re-indexing.
Action (blocking): Confirm the assumption — scan existing on-chain data (or the indexer DB) for payloads that would fail the new checks. A one-liner check: decode all existing payloads through the new code path and verify zero rejections.
Summary of Required Actions
| # | Severity | Action | Blocking? |
|---|---|---|---|
| 1 | P0 | Grep for remaining binary.ReadUvarint calls in pkg/ark-lib/ |
Yes |
| 2 | P1 | File tracking issues for TS/dotnet SDK canonical validation | No |
| 3 | P1 | Doc note on reader position in varint.ReadCanonical |
No |
| 5 | P2 | Comment on definedPresenceMask maintainability |
No |
| 🔒 | Critical | Verify no historical on-chain payloads use non-canonical encoding | Yes |
Requesting changes primarily for the protocol-critical verification (historical payload scan) and the binary.ReadUvarint audit. The code quality is high — this is good hardening work.
🤖 Reviewed by Arkana (arkade-os/arkd code review agent)
pkg/ark-lib has no remaining direct parser use of binary.ReadUvarint; the only real call is inside internal/varint.ReadCanonical, which is the intended wrapper. Both extension and asset varint decode paths are covered by it. |
|
Other issues raised by Arkana can be ignored IMO. |
arkana-ai-bot
left a comment
arkana-ai-bot
left a comment
There was a problem hiding this comment.
Reviewed at head 301e197. This is a well-scoped canonicality tightening; the invariant serialize(parse(x)) == x is a strong anti-malleability guarantee for OP_RETURN payloads and the checks are all O(1) on the decode path. LGTM with a few notes.
Correctness spot-checks
internal/varint/varint.go—ReadCanonicalcorrectly leveragesbinary.ReadUvarintfor overflow handling and comparesbytesConsumed == PutUvarint(v).lento reject non-minimal encodings. Verified against0x80 0x00(0 overlong),0x81 0x00(1 overlong), and multi-group overlongs. Overflow path (10th byte > 1) is delegated correctly.extension/extension.go:210-223— the ordering is right:tokenizer.Next()→tokenizer.Err()→pushOpcode != minimalPushOpcode(len). Trailing non-push opcodes (e.g. OP_NOP, OP_1..OP_16, OP_RESERVED) get rejected as "trailing data after payload" because they're distinct tokens after the payload push. OP_0 as the second op would degenerate topayload=empty, opcode=0x00which matchesminimalPushOpcode(0)=0, but then the magic-prefix check fails downstream. Good.asset/asset_group.go:239-245— presence-bit mask check catches every bit outside0x07. The metadata-flag-with-empty-list check at 274-278 is necessary becausenewMetadataListFromReaderhappily returns an empty list oncount=0.asset/utils.go— bothdeserializeVarUintanddeserializeVarSliceroute throughvarint.ReadCanonical, so every varint inasset_input.go,asset_output.go,metadata.go,packet.go, andasset_ref.goinherits canonicality. Verified nobinary.ReadUvarintcallers remain in the asset/extension packages.- The packet fuzz oracle change from stability-of-round-trip to
data == serialize(parse(data))is strictly stronger — good replacement.
Non-blocking observations
-
Historical-parse assumption.
getAssetsFromTxOutsininternal/infrastructure/db/service.go:726and the intent-proof / ark-tx parsers ininternal/core/application/service.go:1810and:4411re-parse stored on-chain bytes with the new stricter rules. The PR body notes the assumption of no non-canonical historical payloads; worth confirming with the team that no mainnet/testnet indexer state contains a third-party extension that used0x80 0x00-style varints or an OP_PUSHDATA1 push under 76 bytes. If any exist, re-indexing will fail. -
Fuzz seed corpus for both fuzzers is heavy on framing but light on asset-body cases. Consider seeding
extension_fuzz_test.gowith an asset packet whose group has an undefined presence bit (e.g.6a0d41524b0008010800010100000151-ish) and one withmetadata_flag | empty list, so those code paths get corpus-mutated during runs. 11M execs without a counterexample is reassuring but the current seeds don't directly exercise those branches. -
deserializeVarSlice(both copies) usesr.Read(buf)after al > uint64(r.Len())guard. Safe forbytes.Readertoday, butio.ReadFullwould be more defensive if the reader type ever changes. Pre-existing, not introduced here. -
Packet.Serialize()inasset/packet.go:70returns(nil, nil)for empty packets; unreachable viaNewPacketFromBytes(which rejects empty), but a manually-constructedPacket{}will produce nil bytes without error. Minor API smell, not this PR's problem. -
minimalPushOpcodeinextension.goshould stay in lockstep withopReturnScript; consider deriving one from the other or adding a compile-time assertion, in case someone later adjusts the boundaries (75/255/65535). ATestOpReturnScriptMatchesMinimalPushOpcodecovering the boundaries (0, 1, 75, 76, 255, 256, 65535, 65536) would lock the coupling.
None of the above block the PR.
|
⏰ This PR (enforce canonical encoding for Arkade payloads) has been open for 70 days without a review. @msinkec is anyone looking at this? |
arkana-ai-bot
left a comment
arkana-ai-bot
left a comment
There was a problem hiding this comment.
Arkana review — Enforce canonical encoding for Arkade payloads
Good defensive hardening of the asset packet parser.
Changes:
- Undefined presence bits (): rejects any presence byte with bits outside the defined mask (0x07). Prevents a payload with garbage high bits from being accepted as a valid group.
- Empty metadata with flag set (): rejects presence 0x04 (metadata flag) paired with an empty list. Closes the non-canonical form where the flag is set but no metadata follows.
- Non-minimal varint (): switches
deserializeVarUintfrombinary.ReadUvarint(accepts overlong encodings) tovarint.ReadCanonical(rejects them). This is the most impactful change — LEB128 overlong encodings can produce identical values from distinct byte sequences, which could affect txid binding and enable encoding-based bypass of checks that operate on the serialized form.
Fuzz test upgrade: The fuzz corpus property now asserts that any accepted input round-trips to itself (canonical = parse∘serialize is identity). Previously it only checked parse∘serialize∘parse = parse. The stronger property would have caught the overlong-varint case automatically.
Tests: , , — all target the specific non-canonical forms introduced. / verify the happy path is still accepted.
Ready to merge.
arkana-ai-bot
left a comment
arkana-ai-bot
left a comment
There was a problem hiding this comment.
Arkana review · commit 301e197
Canonical-encoding enforcement across the Arkade payload layer. This closes a malleability surface: without it a node accepting non-minimal LEB128 and non-minimal script pushes would compute a different commitment from one that rejects them, and an attacker could craft payloads that parse differently on different nodes.
What the fix does
- Introduces
varint.ReadCanonical(internal package) to replacebinary.ReadUvarintindeserializeVarUintanddeserializeVarSlice— rejects any leading zero continuation byte (non-minimal encoding). asset_group.go: rejects undefined presence bits and themetadata flag set + empty listcase.extension.go: checks that the data-push opcode afterOP_RETURNis the minimal one for the payload length, and that there are no trailing bytes after the payload push.- Fuzz test updated to assert the canonical property: any input that parses must re-serialize to the same bytes.
Test coverage
- Unit tests for undefined presence bits, empty metadata list, overlong group count, non-minimal push opcode, and trailing bytes.
- Canonical roundtrip fuzz property is now enforced rather than just tested for stability.
Protocol impact
Any client/node that submitted non-canonical payloads before this change will now be rejected. Confirm the field has a coordinated upgrade window if non-canonical payloads are known to exist in deployed state.
Looks solid. Ready for human review and merge decision.
|
This PR has been open for 6 days without review. @msinkec is anyone looking at this? |
|
This PR (enforce canonical Arkade payload encoding) has been open 86+ days without review. @msinkec is anyone looking at this? |
arkana-ai-bot
left a comment
arkana-ai-bot
left a comment
There was a problem hiding this comment.
Review: enforce canonical encoding for Arkade payloads
Verdict: looks ready to merge.
What the PR does
Tightens Arkade payload parsing to enforce canonical (minimal) encoding at every layer:
asset_group.go: Rejects undefined presence-byte bits (bits outside the three defined masks) and rejects a metadata presence flag paired with an empty metadata list.utils.go/extension.go: Replacesbinary.ReadUvarintwith a newvarint.ReadCanonicalthat rejects overlong LEB128 encodings.extension.go: After parsing the OP_RETURN payload push, verifies (a) no trailing script bytes and (b) minimal push opcode.minimalPushOpcodemirrorsopReturnScript's encoding logic.- Fuzz corpus update:
FuzzNewPacketFromBytesnow asserts that any accepted encoding is its own serialisation (canonical identity). This is the right invariant — it will catch any future non-canonical-but-accepted encoding.
Test coverage
Good: canonical baseline round-trips, undefined presence bits, empty-metadata flag, overlong varint group count, non-minimal push opcode, and trailing bytes are all exercised. The fuzz harness covers broader input space.
Minor observations
minimalPushOpcodeandopReturnScriptare co-located inextension.go. IfopReturnScriptchanges encoding (e.g. for large payloads),minimalPushOpcodeneeds to track it. A single source of truth would be cleaner long-term, but fine for current scope.- The undefined-presence-bits check (
definedPresenceMask) is a strict gate — future protocol extensions that add presence bits must update it. Worth a note in architecture docs or the constant comment.
No security issues found. The canonical enforcement reduces the malleable input surface. Ready to merge.
|
This PR has been open for 90+ days without a human review. @msinkec is anyone looking at this? Arkana reviewed the canonical encoding enforcement (looks ready to merge). |
|
This PR has been open for 91 days without review. @msinkec — is anyone looking at this? |
|
This PR has been open 94+ days without a formal review decision. @msinkec — reviewed as ready to merge; are there any blockers? |
arkana-ai-bot
left a comment
arkana-ai-bot
left a comment
There was a problem hiding this comment.
Arkana review — 2026-08-30
What this does
Enforces canonical encoding for Arkade payloads at parse time, across three layers:
-
Varint canonicality (): replaces (which accepts overlong LEB128 encodings) throughout and . A non-minimal varint now returns an error instead of silently accepting an alternative encoding of the same number.
-
Asset group presence bits: rejects any byte with undefined bits set (), and rejects the metadata-present flag when the metadata list is empty — both non-canonical states that would round-trip differently.
-
Extension script framing: rejects trailing bytes after the payload push, rejects non-minimal push opcodes ( check), and uses for the per-packet length prefix.
-
Fuzz test strengthened: now asserts (input must equal its own re-serialization), catching any non-canonical input that slips through.
Why this matters
Canonical encoding is a security property for consensus-sensitive data: two nodes parsing the same wire bytes must produce identical parsed structures (and identical txids/hashes), and an encoder that accepts non-canonical input opens the door to encoding-ambiguity attacks.
Correctness
- logic: rejecting encodings where the top 7 bits of the last byte are zero while there are continuation bytes, and rejecting values that don't fit uint64 — matches Go's varint spec for minimal encoding. ✓
- Presence mask check: cleanly isolates undefined bits. ✓
- : covers ≤75 / ≤255 / ≤65535 / larger thresholds, matching Bitcoin script encoding rules. ✓
- Existing test vectors: the baseline canonical round-trip tests verify that nothing valid was accidentally broken. ✓
Tests
Good: unit tests for undefined presence bits, empty metadata with flag set, overlong group count varint, non-minimal push opcode, trailing bytes; all targeted at specific non-canonical forms. Fuzz corpus seeds are preserved.
Minor
No issues.
Verdict
Correct and well-motivated hardening. Ready to merge after a human review.
|
This PR has been open for 5+ days without a review decision. @msinkec is anyone looking at this? |
2 participants
Enforces that the Arkade
OP_RETURNpayload has a single canonical byte encoding, closing encoding-level malleability. The guiding invariant: any payload arkd accepts must re-serialize to exactly the bytes it parsed.Changes
internal/varint.ReadCanonicalrejects non-minimal LEB128, e.g.0x80 0x00; used by the asset and extension decoders.OP_RETURNpush opcode.All checks are O(1) on the decode path, with no re-serialization.
Scope
Encoding canonicality only. List and packet ordering are unchanged: those already round-trip, and asset-group order is semantically meaningful as issuance index.
Testing
parse(x) ok ⇒ serialize(parse(x)) == x; approximately 11M execs each, with no counterexample.Compatibility
arkd already emits canonical bytes, so no legitimate flow is affected. This only tightens acceptance of malformed or third-party input. It assumes there are no non-canonical historical on-chain payloads to re-index.
Summary by CodeRabbit
Release Notes
Bug Fixes
Tests