Skip to content

OpenTelemetry spans and metrics for enrichment jobs - #67

Draft
asg017 wants to merge 3 commits into
otel/02b-empty-jobsfrom
otel/03-telemetry
Draft

asg017 wants to merge 3 commits into
otel/02b-empty-jobsfrom
otel/03-telemetry

Conversation

@asg017

@asg017 asg017 commented Sep 28, 2026

Copy link
Copy Markdown
🤖 Claude-generated PR description

Adds enrichment-specific OpenTelemetry spans and metrics, built on Datasette 1.0a41's plugin telemetry kit.

Stack: lint (#63) → 1.0a41 compatibility (#64) → supervised background tasks (#65) → empty jobs (#66) → telemetry (this PR). The base branch is otel/02b-empty-jobs.

There are three commits, one per step:

  1. Telemetry scaffolding: registry, helpers, kit fixtures
    • opentelemetry-api>=1.37 becomes a runtime dependency; opentelemetry-sdk is dev-only.
    • telemetry_registry.py is the single source of truth for every wire name: 5 spans, 6 metrics, and 14 attributes under enrichments.*, plus core's db.namespace and error.type.
    • telemetry.py holds API-only helpers. It never imports the SDK, and with no provider installed everything is a no-op. It has no schema_url. Spans are always started with record_exception=False and set_status_on_exception=False.
    • Kit fixtures are added to conftest, and the never-imports-the-SDK test runs first.
  2. Trace enrichment runs and batches, record job metrics
    • Each job run is the root of a new trace, enrichments.job.run, linked to whatever started it. It carries trigger=enqueue|resume|restart. What it links to:
      • the enqueue POST span;
      • the resume POST span;
      • the restart span. That span is itself a linked root, so on hosts without lifespan it doesn't hang off a random first request.
    • enrichments.batch is a span per batch (never per row). The internal row fetch nests under it through injected W3C trace headers. That is a workaround for OpenTelemetry: datasette.client requests start a new trace instead of nesting under the caller's span simonw/datasette#2969, and a comment links it.
    • Other spans:
      • enrichments.initialize is a child of the POST;
      • enrichments.finalize is a child of the run;
      • enrichments.restart.
    • Six metrics, with bounded dimensions only (the enrichment slug and enum values):
      • job.runs
      • job.runs.active
      • job.run.duration
      • batch.duration
      • rows
      • cost
    • Run outcomes are finished, paused, cancelled, stopped, interrupted (shutdown) and error. error.type is only ever the class name.
    • start_enrichment_in_process() gains an optional trigger= keyword argument, so existing callers keep working.
  3. Telemetry conformance and privacy tests, docs/telemetry.md
    • Static registry tests: names are pinned, the prefix rule holds, metric dimensions are bounded, buckets are set where needed, and every entry has a description.

    • One workload fires every signal, checked in both directions: everything emitted is declared, and everything declared is emitted.

    • The privacy sentinel test plants unique strings in:

      • config
      • the filter
      • row values
      • the text primary key and cursor
      • the actor
      • the exception message
      • the Pause and Cancel messages
      • a secret

      It then asserts none of them appears in any span or metric. That includes core's spans (no scope_name).

    • docs/telemetry.md:

      • turning it on
      • an example trace tree
      • outcomes
      • privacy, including core caveats: _where literals end up in db.query.text, and failed DB calls record their exception message on core's span
      • a span, metric and attribute reference generated by cog
    • A "Telemetry" section for enrichment authors in developing.md.

    • cog --check docs/*.md added to the CI lint job, and cogapp added to the dev group.

    • Also edits one registry description string, for batch.outcome.

Privacy

The telemetry never records:

  • config, including prompts and secrets
  • filter_querystring
  • row or primary-key values
  • next_cursor
  • actor_id
  • exception messages or tracebacks
  • Pause/Cancel messages

This is enforced by the sentinel test. Independent reviews confirmed zero leaks across all scopes.

Testing

  • 62 passed, including under -W error::pytest.PytestUnraisableExceptionWarning.
  • Stable across repeated runs, under CPU contention and with shuffled test order. It also passes on Python 3.10.
  • Ruff and cog are clean, and the Sphinx build has no warnings.
  • Break-it checks: each of these was done and then reverted.
    • Renaming a span in the registry fails test_wire_names_are_pinned.
    • Adding a span attribute holding json.dumps(config) fails the privacy test.
    • Removing finalize_span fails assert_spans_covered.
    • Removing the trace-header injection fails test_batch_spans_are_children_of_run.

🤖 Generated with Claude Code

asg017 and others added 3 commits September 28, 2026 14:01
Declares every span, metric and attribute in telemetry_registry.py using
Datasette 1.0a41's plugin telemetry kit. No instrumentation yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each run is a new trace linked to the request (or restart) that started
it, with a child span per batch. Adds run/batch/row/cost metrics.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Registry conformance via Datasette's telemetry_testing kit, sentinel
privacy checks, and a cog-generated span/metric reference checked in CI.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant