scheduler: run duppy on a pool of domains, retire its monad, and stop rescanning the event loop

[toots]

Mirror of savonet/liquidsoap#5359

Duppy queues were systhreads, and on OCaml 5 those all share domain 0's runtime lock, so they gave concurrency and no parallelism — adding queues subtracted throughput.

Duppy now spawns one dispatch domain per core plus one for the event loop, batching non-blocking tasks onto a domain and handing blocking ones an auxiliary thread inside it, so parking in a syscall frees the domain.

With domains in place the CPS monad OCaml 4 required is no longer needed: computations park on an effect and resume where they left off, so harbor, the telnet server and harbor output read and write their sockets as ordinary sequential code.

The event loop no longer rescans what it is waiting on. Descriptors are watched through a set the kernel keeps — epoll on Linux, kqueue on BSD, select where neither exists — and tasks are found by descriptor and by deadline instead of by folding over all of them.

Breaking: settings.scheduler.generic_queues, fast_queues and non_blocking_queues are removed — they counted threads that no longer exist — and settings.scheduler.blocking_tasks replaces them to bound blocking tasks; scripts setting the old ones must drop those lines. Requires OCaml 5.5.

Measurements

8 cores, OCaml 5.5.0.

Parallelism. Scheduled work runs on every core instead of taking turns on one:

	before	after
32 CPU-bound tasks, 1 → 6 domains	—	3.04s → 0.449s (6.8x)
16 CPU-bound tasks, 1 → 8 queues	3.58s → 5.11s	—

Streaming latency. A 20ms periodic thread — a clock's shape — measured against CPU-bound scheduler work. The old medians are one 50ms systhread tick per contender:

busy tasks	before	after
2	p50 53.9ms	p50 0.09ms
4	p50 100ms	p50 0.07ms
7	p50 177ms	p50 0.06ms

Waiting for descriptors. poll rescans everything submitted to it; a kernel-held set does not:

descriptors watched	poll	Pollset.wait
500	37.1 µs	0.5 µs
2000	107.8 µs	0.3 µs
4000	233.7 µs	0.3 µs

Cost of one loop turn, against the number of tasks waiting on quiet sockets — the shape of a harbor with that many connected-but-silent clients:

waiting tasks	before	after
0	10.4 µs	~40 µs
500	43.4 µs	~40 µs
1000	79.5 µs	~40 µs
2000	168.9 µs	~40 µs

Flat rather than linear, which is the point. The after column is one figure because it no longer varies; it is noisy between 31 and 50 µs because cross-domain dispatch now dominates the loop entirely, and it is worse than the old empty case for the same reason — the two are not measuring the same path once the loop stops being the bottleneck.

Degraded cases

A flood of non-blocking tasks on one domain used to starve everything else. Taking every ready immediate task on every round left blocking work unreachable whenever the ready list refilled as fast as it drained: 200 self-rearming tasks on one domain ran 2.6 million times in two seconds while a waiting blocking task never ran once. At four domains it was invisible, because the worker taking the batch hands the leftover blocking work to another worker and with one worker there is no other worker. A worker that just took a batch now takes a blocking task next when it has one and is under its cap, which bounds the wait to one batch per worker.

Batch length is still unbounded by design, so a task classified Immediate that does not return immediately is a latency bug the scheduler will not rescue.

Clocks are unaffected by scheduler load, including on one core. They stay on domain 0 rather than joining the pool, so they never queue for a dispatch slot and compete only for CPU — arbitrated by the OS across domains instead of by OCaml's 50ms tick within one. Pinned to a single core, the periodic thread holds p50 0.03–0.08ms with up to 7 busy tasks.

Deletions

duppy.ml 1141 → 800 lines, duppy.mli 512 → 255, and duppy_stubs.c gone entirely: its only function backed ba_write, which the sole harbor transport implemented as failwith "Not implemented!" and nothing called. Duppy.Monad.Mutex and Condition went with Server.condition, which was exported but had no callers anywhere.

Not here

Clocks keep their own threads and their own high-precision wait — they park inside Pcm.writei and a JACK semaphore, where no continuation can be captured, and the event loop's timer is coarser than the clock_nanosleep they use today.

Tests

src/modules/duppy/test/test_duppy.ml covers cross-domain dispatch, batching, socket events, the blocking cap, draining on stop, effect resumption landing on a different domain than it started on, and the starvation case above. src/modules/stdlib-utils/test/test_pollset.ml covers registration, readiness, level-triggering, timeouts, peer close, and the property the kernel-held set exists for: with 500 watched and one ready, a wait returns exactly one. LIQ_POLLSET_BACKEND=select forces the fallback, so the backend Windows runs is exercised on Linux too — both it and epoll pass the same tests, and duppy's own suite passes on both.

Each check was falsified by reverting the fix and watching it go red. Harbor 5/5, language and regression tests pass, and the suite runs clean pinned to one core and under full CPU load. The telnet server, which has no automated test, was driven by hand.

kqueue compiles nowhere reachable from here and is CI-only.

Also fixes the logging thread dropping everything logged during shutdown, type names coming out empty in errors raised from a callback, and time.zone.set silently doing nothing once anything had read the local time.