Skip to content

fix(client-cohttp-eio): interruptible export waits at shutdown - #5

Open
dijkstracula wants to merge 2 commits into
forkfrom
nathan/otel-interruptible-shutdown
Open

fix(client-cohttp-eio): interruptible export waits at shutdown#5
dijkstracula wants to merge 2 commits into
forkfrom
nathan/otel-interruptible-shutdown

Conversation

@dijkstracula

@dijkstracula dijkstracula commented Aug 11, 2026

Copy link
Copy Markdown

Problem

When an OTLP collector cannot be reached during a traced run, exiting the process takes several seconds. Two things combine:

  1. The core function Collector.remove_backend calls B.tick() before calling B.cleanup(). As a result, during shutdown an export with force:false runs while the flag stop is still false; stop is only set in cleanup.
  2. send_http waits for 3 seconds (Eio_unix.sleep 3.) when an export fails.

Because this failed POST blocks for the whole backoff period before cleanup runs, the loop cannot finish. Changing the sleep to be interruptible alone doesn't solve it, because no broadcast occurs while the tick is blocked; the signal that would stop the loop lives in cleanup, which happens after the blocked tick.

Fix

Move the backoff out of the sending logic so that an export, and thus a tick, never blocks on it:

  • send_http now fails fast and has a time limit (send_timeout_s). If it fails, it does not sleep. The timeout also limits a black-hole endpoint (accepts the connection but never replies), which before had no timeout.
  • The ticker controls the backoff as its wait between ticks. Both the normal cadence and the post-failure backoff compete with a one-shot shutdown promise (wait_or_shutdown). This means shutdown can interrupt them immediately.
  • The ticker remains a non-daemon fiber, so an in-flight final export is awaited instead of being cancelled mid-send (which would drop a batch it had already popped). The forced flush in cleanup is wrapped with Cancel.protect.

No change to core Collector.remove_backend, and no daemon fiber required.

Validation

Measured with a standalone harness and mock collector for three cases: refused, reachable, and black‑hole:

scenario before after
unreachable (connection-refused), shutdown mid-backoff ~3000 ms (deployed ade04b2) 0.9 ms
reachable, final batch flushed only at cleanup (force:true) delivered delivered (no loss)
black-hole (accepts, never replies) unbounded bounded at send_timeout_s

Relationship to #4

This approach is an alternative to #4. While #4 removes the backoff and daemonizes the ticker, our fix keeps the backoff but makes every wait interruptible. Therefore we don't need a daemon or changes to final batch flushing. The shutdown latency for connection‑refused remains the same, and we also bound the black‑hole case.

Note: Because remove_backend runs a tick before cleanup in core, this stall also affects main, upstream, and the ocurl/lwt backends. We plan an upstream report and PR as a follow-up.

🤖 Generated with Claude Code

dijkstracula and others added 2 commits August 11, 2026 09:45
…ing the backoff

Shutdown could stall ~3s: Collector.remove_backend runs one tick before cleanup
while [stop] is still false, and send_http's failed-export backoff was a plain
Eio_unix.sleep, so that pre-cleanup tick blocked out the whole backoff before
cleanup (which sets stop and would interrupt it) could run.

Move the backoff out of the send path:
- send_http is fail-fast and time-bounded (send_timeout_s); no export, and thus
  no tick, ever blocks in a backoff, so the pre-cleanup tick cannot stall.
- the ticker owns the backoff as its inter-tick wait; both the cadence and the
  backoff wait race a one-shot [shutdown] promise, so shutdown interrupts either.
- the ticker stays a non-daemon fiber so its in-flight final export is awaited,
  not cancelled mid-send; cleanup's forced flush is Cancel.protect'd.

Needs no core Collector.remove_backend change and no fork_daemon.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The semgrep org ruleset rejects any action not pinned to a full-length commit
SHA, so the fork's mutable refs (checkout@main, checkout@v4, setup-ocaml@v3,
install-nix-action@v30, actions-gh-pages@v3) failed every job at startup. Pin
each to the SHA main already uses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dijkstracula
dijkstracula force-pushed the nathan/otel-interruptible-shutdown branch from f92a019 to b54c883 Compare August 11, 2026 16:45
@dijkstracula dijkstracula changed the title fix(client-cohttp-eio): interruptible export waits at shutdown (keep the backoff) fix(client-cohttp-eio): interruptible export waits at shutdown Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant