arnm

C primitives for embedding. Data lives in Memory-Arenas.

A small C11 library meant to be linked into a program written in something else — Node.js, Go, Rust, C++. It brings a bump allocator, containers built on it, and the conversions a host does constantly and the standard library does slowly. Everything it hands out comes from an arena, and the arena comes from you: arnm brings no memory policy of its own — you give it a blob, it works inside that blob, and it gives the blob back.

#include <stdalign.h>
#include "arnm/arena.h"

alignas(8) uint8_t blob[64 * 1024];   // wherever this comes from: the host decides
arnm mem;
arnm_init_arena_borrow(&mem, blob, sizeof(blob));

uint8_t *buffer = NULL;
arnm_alloc(&buffer, 128, &mem);
/* ... */
arnm_free(buffer, 128, &mem);

arnm_reset(&mem);              // the whole arena, in one move

A borrowed buffer is taken exactly as it is. Both halves of that line have to hold: blob must start on an 8 byte boundary, which is what alignas(8) is for, and sizeof(blob) must be a multiple of 8. Neither is rounded — anything else comes back as ARNM_ERROR_INVALID_PARAM and no arena is set up.

The size is the half that surprises people, because the owned arena does round it: arnm_init_arena(&mem, 100) reserves 104 bytes and succeeds, while arnm_init_arena_borrow(&mem, blob, 100) is refused. The difference is who owns the memory — rounding 100 up to 104 in a buffer you sized at 100 would let the arena hand out four bytes past its end, so the size is questioned instead of corrected. Declare the blob at a multiple of 8 rather than trimming the call. arnm_multi_arena_borrow() borrows under the same rule.

Pass NULL instead of an allocator and every call falls back to malloc/free. That is the only place in the library where malloc appears — lint.sh fails the build if a second one shows up. The JSON parser vendored under third_party/ is held to the same line: it is handed an allocator that forwards to arnm_alloc/arnm_realloc/arnm_free, so its own default allocator is on no path this library takes.

What is in it

Header	What it gives you
arnm/memory.h	the arnm handle and the calls every allocator answers: alloc, free, realloc, clone, reset
arnm/arena.h	make a handle an arena -- over memory it takes from the host, or memory you lend it
arnm/memory_block.h	pointer and size kept together, so freeing needs no bookkeeping from you
arnm/byte_buffer.h	one block filled from the front, records packed back to back; handed to a stream in one piece
arnm/multi_arena.h	a chain of arenas that opens another one instead of running dry
arnm/fixed_arena_pool.h	a fixed set of equal sized arenas, lent out and returned; the peak is known at init
arnm/graded_block_pool.h	blocks in power of two grades on a free list per grade, from a chain of arenas the pool owns; what a growing container outgrows serves its next request instead of staying behind in an arena
arnm/bucket_vector.h	growing sequence with stable element addresses; no copy on growth
arnm/hash.h	a fast hash and SipHash-1-3, both over every byte and inline, so a length known at compile time unrolls the loops
arnm/key_map.h	fixed size keys to dense ids 0, 1, 2, ...; linear probing over a hash of every key byte -- a fast one, or SipHash under your key -- the keys stored once beside the table; ARNM_KEY_MAP_DEFINE inlines the lookup for one key size
arnm/roaring_bitmap.h	compressed set of uint32_t values added in ascending order; sorted array while small, roaring containers once it grows, blocks from a graded block pool
arnm/roaring_ops.h	and, or, andnot and a range copied out of such sets, each into a set you provide
arnm/roaring_query.h	how many values several such sets match, and which page of them -- in all of these, one of those, none of these, inside a range -- without building a result
arnm/fixed_ring.h	bounded queue, first in first out; the room taken once, a full ring refuses rather than grows
arnm/json_reader.h	JSON parsed into your arena; one line per struct field, the first error kept with its field name, an in-situ path that copies nothing
arnm/json_writer.h	the way back: one line per struct field, strings and keys borrowed rather than copied, and the text rendered straight into the allocator you name
arnm/utf8.h	is this run of bytes UTF-8, and where does it stop being it -- the check the reader does for you and the writer does not
arnm/converter.h	integer to decimal string, roughly 4× faster than snprintf; bytes to lowercase hex and back, uuid to its 8-4-4-4-12 form and back
arnm/duration.h	nanoseconds to a readable span
arnm/mono_timer.h	monotonic clock, one type, three units
arnm/result.h	one result code for everything, with a range reserved for you

The contract

• Sizes are uint32_t. Counts, indices, byte sizes. Anything that would not fit returns ARNM_ERROR_ARITHMETIC_OVERFLOW instead of wrapping.
• Sizes are passed in, never stored. Freeing and resizing need the size you allocated with. arnm_memory_block keeps the two together when you would rather not.
• Every size rounds up to 8, so every pointer the arena hands out is 8 byte aligned.
• An arena only takes back its most recent allocation. Anything before it stays until arnm_reset. Calls that could not reclaim return ARNM_WARNING_ARENA_MEMORY_NOT_RECLAIMED — the operation happened, the memory did not come back. It is neither a failure nor a release; handle it where it appears.
• Failures leave every output untouched.

Asking an arena where it stands

Two figures, read from opposite sides of the same line, and both a field read rather than a walk:

uint32_t left = arnm_arena_remaining(&mem);   // what is still there
size_t   over = arnm_arena_overflow_total(&mem);  // what was already turned away

arnm_arena_remaining() is the largest request that would still be served -- an arena never reaches past the block it was given. It is asked before a request, which is what makes it useful: code with somewhere else to put a large block can look first instead of being refused to find out. The figure is always a multiple of 8, and it holds until the next call that moves the index.

arnm_arena_overflow_total() is the other half, and it is only ever read afterwards. Every request the arena had to refuse adds its rounded up size, saturating at UINT32_MAX rather than rolling over. Run the real workload once and the figure says how much larger the arena would have had to be — the sizing question a guess only postpones. arnm_reset() and arnm_release() clear it.

Both answer 0 for NULL, for host mode and for a chain, so a 0 from either is "full" only when the handle really is a single arena. arnm_is_arena() and arnm_is_multi_arena() tell those cases apart, and a chain is asked through arnm_multi_arena_measure() instead — it opens more ground rather than running dry, so one remainder could not say what it appears to.

When one arena is not enough

An arena has the capacity it was born with. A chain keeps a set of them and opens the next one when the current stretch fills, so the size never has to be guessed right up front — and a request larger than the arena capacity gets ground of its own instead of a refusal.

A chain is an arnm * like any other allocator: it is built through this header and then used through the calls above.

#include "arnm/multi_arena.h"

// 1 MiB per arena, and an arena drops out of the search once under 4 KiB is left.
// {0} would take every default: 1 KiB arenas, a 64 byte threshold, no ceiling on the count.
arnm_multi_arena_options options = {0};
options.arena_capacity = 1024 * 1024;
options.full_remaining = 4096;

arnm *chain = arnm_create_multi_arena(&options, NULL);
arnm_multi_arena_borrow(chain, blob, sizeof(blob));  // optional: lend it the host's blob,
                                                     // same alignas(8) and multiple-of-8 rule

uint8_t *buffer = NULL;
arnm_alloc(&buffer, 4096, chain);   // the same calls a single arena takes
arnm_free(buffer, 4096, chain);

arnm_reset(chain);                  // every allocation, in one move; the arenas stay
arnm_multi_arena_shrink(chain);     // and give the empty ones back to the host
arnm_destroy(chain, NULL);          // naming the allocator it was created with

arnm_alloc, arnm_free, arnm_realloc, arnm_clone and arnm_reset do not care which of the two they were handed. What stays chain specific is what only a chain can answer: _reserve(), _borrow(), _shrink(), _arena_count() and _measure() — the last of which reports bytes reserved against bytes used, and how much of the chain the search still walks.

NULL returned from arnm_create_multi_arena() means the options were refused or the allocator had no room; arnm_multi_arena_options_validate() tells the two apart, and fills a set of options in with the defaults it would really run on. The allocator passed there carries the chain's own bookkeeping only — the arenas always come from the host, because a chain drawing them from a fixed allocator could not outgrow it, which is the one thing it exists to do.

Pointers stay put: an arena, once opened, is never moved or resized. A borrowed block stays the host's: the chain fills it and never frees it. Set arena_max_count to cap the chain, and it answers ARNM_ERROR_RESOURCE_EXHAUSTED instead of opening another arena — a bounded budget with a known peak.

full_remaining is the field worth thinking about. A request is served first fit, and an arena leaves that search for good once its remainder falls to the threshold. So the question it answers is: what is the smallest request that should still land in a leftover? An arena holds a request of n bytes while at least n are left, and is written off at or below the threshold, so n - 8 drops it out of the search exactly when it can no longer take that request.

For uniform records that is the whole story — one alignment step below the record size wastes nothing and keeps the search short. For mixed sizes it is a trade: lower gives up nothing usable but leaves arenas in the search holding remainders only the small requests fit, at around half a nanosecond per arena walked, which only bites once a thousand of them have piled up; higher keeps the search short and writes off up to a threshold worth of bytes per arena. bench_multi_arena puts numbers on both ends.

Reading JSON

arnm/json_reader.h parses a document into the allocator you name and reads it back one field per line. The parser is yyjson, copied into third_party/ at release 0.12.0 — it is compiled into libarnm and never surfaces: the header is plain C11, and you link one library.

The reader keeps the first error and the field it happened at, so a struct is filled without a test between the lines and asked about once, at the end:

#include "arnm/json_reader.h"

arnm_json_reader reader;
arnm_json_reader_init(&reader, &mem, ARNM_JSON_READ_DEFAULT);  // NULL is the host, as everywhere
arnm_json_reader_parse(&reader, text, length);

config.host    = arnm_json_reader_get_string(&reader, "host");
config.port    = arnm_json_reader_get_uint32(&reader, "port");
config.timeout = arnm_json_reader_get_double(&reader, "timeout");
config.debug   = arnm_json_reader_get_bool(&reader, "debug");

if (ARNM_SUCCESS != arnm_json_reader_status(&reader)) {
  log("%s at '%s'", arnm_result_to_string(arnm_json_reader_status(&reader)),
      arnm_json_reader_error_field(&reader));
}

arnm_json_reader_release(&reader);              // every borrowed string dangles from here on

Once something is refused, every later getter answers its empty value — NULL, false, 0 — and leaves the recorded error alone. A refusal in the middle of a struct therefore does not have to stop the reading, and arnm_json_reader_clear_error() picks it up again where you want to go on deliberately.

Nesting is two lines and no bookkeeping: arnm_json_reader_enter() steps into a member and hands back the value it left, arnm_json_reader_leave() puts it back. Arrays go by arnm_json_reader_enter_at() with arnm_json_reader_count() as the bound, and the reader remembers where it stood, so walking one costs a step per element rather than a walk per element. A key of NULL names the current value itself, which is how the elements of an array of scalars are read.

arnm_json_value *array = arnm_json_reader_enter(&reader, "peers");
for (uint32_t i = 0, n = arnm_json_reader_count(&reader); i < n; ++i) {
  arnm_json_value *element = arnm_json_reader_enter_at(&reader, i);
  config.peers[i].name = arnm_json_reader_get_string(&reader, "name");
  config.peers[i].port = arnm_json_reader_get_uint32(&reader, "port");
  arnm_json_reader_leave(&reader, element);
}
arnm_json_reader_leave(&reader, array);

A string a getter hands back points into the document and dies with it. Naming an output allocator — arnm_json_reader_set_output_allocator(&reader, &strings) — copies each one there instead, NUL terminated, so it outlives the document, the reader, and the buffer an in-situ parse read through. Nothing else is copied; numbers and bools travel by value already.

The arena that receives those copies can be sized before it is built. Both measurements walk the document's value array rather than its tree — a flat run, no recursion — and answer the exact byte count, terminators and eight byte rounding included:

uint32_t bytes = arnm_json_reader_output_size(&reader);              // every string in the tree
static const char *const wanted[] = {"host", "name"};
bytes = arnm_json_reader_output_size_for_keys(&reader, wanted, 2);   // only what you will read

The first is the cheaper call and the larger answer: a document carrying members you never read pays for them. The second compares every member name against the list, at every depth, so an array of objects is covered by naming its member once — and it reserves only what a mapper will actually copy. Measured in bench_json on a document of 61 values, five of whose members are read and twenty-five of which are not: the full walk takes about 62 ns and reserves 1200 bytes, the keyed walk takes about 148 ns and reserves 200. Both sit well under the parse that precedes them, which costs about 395 ns for the same document.

A read is refused rather than rounded: arnm_json_reader_get_int64() on 3.5 records ARNM_ERROR_ARITHMETIC_OVERFLOW instead of handing back 3, and reading a string as a number records ARNM_ERROR_INVALID_ENUM_TYPE. Under the field level sit the value level calls the same document is reachable through — arnm_json_object_get(), arnm_json_read_uint32(), arnm_json_array_iter_next() and the rest — each answering an arnm_result of its own, for code that wants every step checked where it happens.

null is the one JSON type a table entry cannot ask about: every other type is named by the entry that reads it, but null is the member saying it has no value at all, so a typed entry refuses it and the rest of the walk goes unread. arnm_json_read_is_null() is how a mapper tells "absent" from "there and explicitly nothing" — take the member as a handle with ARNM_JSON_FIELD_VALUE(), ask, and name its type only once the answer says there is one.

arnm_json_reader_parse_insitu() is the cheaper path. It unescapes strings inside your own buffer instead of copying it, which leaves a document at exactly one allocation — and behind an arena that one sits at the tail, so releasing gives every byte back. The price is that the buffer is modified, has to outlive the document, and has to carry ARNM_JSON_READER_INSITU_PADDING spare bytes past the JSON. The copying arnm_json_reader_parse() asks none of that and costs one input copy, which stays buried in an arena until arnm_reset().

Writing JSON

arnm/json_writer.h is the reader read backwards, and keeps its three habits: one line per field, strings borrowed rather than copied, and one check at the end.

#include "arnm/json_writer.h"

arnm_json_writer writer;
arnm_json_writer_init(&writer, scratch, ARNM_JSON_WRITE_DEFAULT, NULL);   // or ..._PRETTY

arnm_json_writer_add_string(&writer, ARNM_JSON_WRITER_KEY("host"), config.host, host_length);
arnm_json_writer_add_uint64(&writer, ARNM_JSON_WRITER_KEY("port"), config.port);
arnm_json_writer_add_bool(&writer, ARNM_JSON_WRITER_KEY("debug"), config.debug);

arnm_memory_block text;
if (ARNM_SUCCESS == arnm_json_writer_write(&writer, output, &text, NULL)) {
  send(text.data);                          // NUL terminated
  arnm_memory_block_free(&text, output);
}

No begin is needed for an object root; the first field opens one. arnm_json_writer_begin_array() opens an array instead, and either one starts the next payload through the same writer. A writer carrying an error refuses to write and answers that error, so the result above stands in for a check after every field — arnm_json_writer_status() and arnm_json_writer_error_field() are there when you want the verdict earlier.

Every key goes in with its own length, and says whether it needs escaping. The writer never walks a key — not for a terminator, and not for a character JSON cannot hold literally. In a mapper the key is a literal whose length the compiler already knows and whose bytes are a plain name, so both passes that nothing asked for simply stop happening. ARNM_JSON_WRITER_KEY() spells the three arguments out of a literal:

arnm_json_writer_add_uint64(&writer, ARNM_JSON_WRITER_KEY("port"), config.port);
arnm_json_writer_add_uint64(&writer, "port", 4, false, config.port);      // the same thing
arnm_json_writer_add_string(&writer, name, name_length, true, value, value_length);

Both are taken at their word and neither is checked against the key: a wrong length is a wrong name in the document rather than a refusal, and false for a key that does hold a quote writes a name the far side cannot parse. ARNM_JSON_WRITER_ESCAPE_KEY() is the macro for that case.

arnm_json_writer_add_string() keeps the pointer it is given and nothing else — no copy, no escaping pass. Every string and every key therefore has to stay where it is until the write, which is what makes serialising a struct cost almost nothing: the payload is read once, at the end, straight out of your own memory. The same rule as for keys applies to the escaping: a value from source is fine as it stands, a value from input needs the flag, or it writes a document the far side cannot parse.

arnm_json_writer_add_string_flags() is where any of those three defaults changes:

arnm_json_writer_add_string_flags(&writer, ARNM_JSON_WRITER_KEY("note"), note, note_length,
                                  ARNM_JSON_WRITER_STRING_COPY |     // will not stand still
                                  ARNM_JSON_WRITER_STRING_ESCAPE);   // came from input

ARNM_JSON_WRITER_STRING_RAW is the third: bytes that are already JSON — a cached fragment, a number formatted by hand — laid into the text unquoted, unescaped and unchecked. It also decides what the serializer reserves for the field, one byte a character against the six a quoted string is charged in case every one of them escapes, which is why the hex and base64 adders write raw.

Nesting is an open and a close, and the writer keeps the levels itself — up to ARNM_JSON_WRITER_MAX_DEPTH, past which it records ARNM_ERROR_RESOURCE_EXHAUSTED rather than reaching for memory mid-field. A key of NULL means "an element of the current array".

arnm_json_writer_open_array(&writer, ARNM_JSON_WRITER_KEY("peers"));
for (uint32_t i = 0; i < config.peer_count; ++i) {
  arnm_json_writer_open_object(&writer, NULL, 0, false);
  arnm_json_writer_add_string(&writer, ARNM_JSON_WRITER_KEY("name"), peer->name, peer->name_length);
  arnm_json_writer_close(&writer);
}
arnm_json_writer_close(&writer);

The text is rendered once, straight into the allocator you name. write() hands that allocator to the serializer and lets it grow there as it goes, so nothing is written to a scratch buffer and copied out; what comes back is the buffer it was built in, shrunk to exactly what the text needed before it is handed over.

That is what replaced the exact running length the writer used to keep. Measuring every field as it went in cost a table lookup per string byte, on every string, whether anyone ever asked for the number — and once the serializer writes into the caller's allocator directly, nothing needs the number to be exact. arnm_json_writer_buffer_size_min() is what is left: a guess from the element count and the layout, free to ask, good for opening an arena at roughly the right size. It is not a bound in either direction — one long string or one large hex blob runs past it — so never size a fixed buffer by it. A caller that has to know the length writes the text and reads it back.

bench_json prices this against the work it precedes, and runs every payload through the reader and the writer alike so the two directions can be read against each other.

Build

zig build                                                    # the static library, host target
zig build -Dtests=true -Dbenchmarks=true
./run_all.sh                                                 # run everything in zig-out/bin

Options: -Dtarget= to cross compile, -Dshared=true for a dynamic library (what a language binding usually wants), -Dsanitize=undefined_behavior or -Dsanitize=thread, and -DsingleOutputDir=true to drop the artifacts without the bin/ and lib/ split.

Targets verified to build: x86_64-linux-gnu, x86_64-linux-musl, x86_64-windows-gnu, x86_64-macos, aarch64-macos. The -msvc ABI targets need the MSVC SDK headers present on the machine — zig does not ship them, so that combination is untested here.

Naming a target also side-steps an incomplete host toolchain: without -Dtarget zig compiles against the system headers in /usr/include, with one it uses the headers it ships itself. If a plain zig build -Dtests=true fails on something like a missing asm/errno.h, install your distribution's kernel headers or pass -Dtarget=x86_64-linux-gnu.

That gap is what CMakeLists.txt is for: the MSVC ABI on Windows, and nothing else. build.zig stays the master build — CMake mirrors it and never leads it.

cmake -B build -DENABLE_TESTS=ON -DENABLE_BENCHMARKS=ON
cmake --build build --config Release
ctest --test-dir build -C Release

ENABLE_TESTS, ENABLE_BENCHMARKS, ENABLE_SANITIZERS and ENABLE_THREAD_SANITIZER are the CMake spellings of the -D options above. AddressSanitizer is the one thing this route offers that zig cannot — zig ships no asan runtime, MSVC does.

Benchmark numbers need -Doptimize=ReleaseFast. In a debug build the hand written digit loop loses to snprintf by a factor of two, because libc ships optimised and your build does not. With optimisation the picture turns around:

unsigned to string        snprintf   61.8 ns      arnm  16.3 ns
signed to string          snprintf   58.2 ns      arnm  13.8 ns

Using it from another zig project

zig fetch --save git+https://github.com/gradido/arnm#v0.8.3

const arnm = b.dependency("arnm", .{ .target = target, .optimize = optimize });
lib.linkLibrary(arnm.artifact("arnm"));
lib.addIncludePath(arnm.path("include"));

Nothing else to check out. zig fetch takes the repository tree and nothing under it, so everything arnm compiles has to be in that tree — which is why yyjson is a copy of two files under third_party/ and not a submodule. It was one until 0.7.2, and a project that depended on arnm got the JSON half as a missing header three layers down.

Changes

CHANGELOG.md -- what moved in each release, newest first.

License

Apache 2.0.

third_party/yyjson is MIT, and carries its own LICENSE beside the two files it consists of. It is the only third party code in the tree, and it is compiled into libarnm, so a binary built from this repository carries both notices.