feat(debugger): report guardrail telemetry metrics - #10152
Conversation
github-actions
Bot
added
semver-minor
debugger
|
Overall package sizeSelf size: 8.84 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.4.0 | 127.33 kB | 447.04 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10152 +/- ##
========================================
Coverage 98.65% 98.65%
========================================
Files 1005 1007 +2
Lines 157435 157793 +358
Branches 74 74
========================================
+ Hits 155312 155670 +358
Misses 2123 2123
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
BenchmarksBenchmark execution time: 2026-09-07 10:53:08 Comparing candidate commit 12d404b in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2307 metrics, 6 unstable metrics.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ce71ea9034
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ce71ea9 to
c178928
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c17892892a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9431a7bf79
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e7b0a8fde
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Add the three Dynamic Instrumentation guardrail metrics required by the
"Debugger Observability for GA" specification, in the `live_debugger`
telemetry namespace shared with the other tracers:
- `events.skipped{reason,event_type}` for probe hits stopped before an
event was created (`rateLimitProbe`, `rateLimitGlobal`)
- `events.dropped{reason,event_type}` for complete events discarded
before transport (reason codes only; the drop paths land separately)
- `capture.incomplete{reason,event_type}` for captures where a limit was
enforced (`runtimeError`, `timeout`, `depth`, `fieldCount`,
`collectionSize`, `stringLength`, `payloadTooLarge`, `other`), counted
once per event per reason
Rate limits are decided by the probe sampler inside the V8 breakpoint
condition on the main thread, while capture limits are enforced in the
debugger worker. To keep both hot paths cheap, counters live in a
SharedArrayBuffer incremented with atomics, and the main thread drains
them into telemetry metrics every 10 seconds and when the debugger is
stopped.
Also report `dynamic_instrumentation.enabled` in the telemetry products
payload, as required by the same specification.
The flush interval timer is unref'ed and the debugger worker does not keep the process alive, so when an application exits on its own everything counted since the last 10 second tick was lost: `stop()` is never called in that case, and telemetry sends its final metrics from its `beforeExit` handler before the timer fires again. For a short-lived process that could be every counter. Telemetry publishes the `datadog:telemetry:app-closing` channel right before that final send, precisely so listeners can update their metrics first. Drain the counters on that channel while the debugger is running.
`Worker#terminate()` only interrupts the worker asynchronously, so a counter the worker increments between the final drain and its actual termination is lost. Those counters can only describe events that are still in the worker's upload buffer, which are discarded together with the worker, so a drain ordered after the worker's exit would not report anything an observer could act on. Spell that out next to the drain.
The paused handler treated every fatal capture error as a runtime error when tagging `capture.incomplete`. The collector however also raises a fatal error on purpose when an object exceeds the large object safety threshold, to disable future captures for the probe. Nothing threw in that case; the object was trimmed to the field count limit, which the processor already reports. The event was counted as both `fieldCount` and a bogus `runtimeError`. Record the runtime error reason where the errors originate instead: when a scope cannot be read, when a capture expression throws and when it cannot be evaluated at all. The safety threshold keeps disabling the capture but is reported as the limit it is.
Messages longer than 8 KiB are cut down before the event is queued for
upload, which is a limit enforced by the tracer just like pruning an
oversized snapshot is, but unlike the pruning it was not reported. Record
it as the string length limit so `capture.incomplete{stringLength}`
covers log events whose message was trimmed.
A scope with more variables than the field count limit is trimmed by
the collector like any other object, which marks the returned array
with the field count symbol. The scope itself is not part of the
snapshot though: its variables are spread into the local state and the
marker was lost with the array, so the extra variables were dropped
without `capture.incomplete{fieldCount}` being reported.
Record the limit when the collector returns a trimmed scope.
A property was processed first and redacted afterwards, so any limit enforced while processing its value (a string trimmed to the maximum length, an object cut off at the reference depth, ...) was recorded as an incomplete capture reason, although the whole value is replaced by the redaction marker and none of those limits show in the snapshot. Decide on the redaction first, like the map entry path already does, and process a redacted value without recording the limits it hits.
Two of the specs replace `process.hrtime.bigint` with a stub that advances the clock by 50 ms on every call, and only the fake timers were restored afterwards. Any spec collecting a snapshot later in the same process then ran into the capture deadline immediately, which now bites the pause handler spec covering the large object safety threshold.
Reaching the capture deadline was recorded as an incomplete capture unconditionally, on top of the timeout markers the collector puts on the nodes it skips. The markers are what the processor reports, and they are ignored for a value that ends up redacted, so when the deadline was reached inside the only value of the last scope and that value was redacted, the metric still reported a timeout the snapshot does not show. The unconditional bit was there for the scopes remaining after the deadline, which are dropped without any marker. Only record it when such scopes exist.
Same as for the local state: reaching the deadline while evaluating capture expressions was recorded unconditionally, although the nodes cut within the expression being collected carry the collector's timeout marker and are recorded when processed, unless they end up redacted. Only the expressions skipped after the deadline are returned as bare stubs without any marker, so only record the timeout for those.
…line Compare against the last collectable scope instead of scanning the rest of the chain with an index based loop and a second helper. Also let the snapshot test util trigger a breakpoint with any function, so the incomplete capture spec no longer needs its own copy of the helper.
eec56f3 to
727ac7c
Compare
Co-authored-by: Cursor <cursoragent@cursor.com>
2 participants
What does this PR do?
Adds three debugger guardrail metrics in the
live_debuggertelemetry namespace:events.skipped{reason,event_type}- probe hits stopped before an event was created (rateLimitProbe,rateLimitGlobal)events.dropped{reason,event_type}- complete events discarded before transport (reason codes only; the drop paths land in the next PR)capture.incomplete{reason,event_type}- captures where a limit was enforced (runtimeError,timeout,depth,fieldCount,collectionSize,stringLength,payloadTooLarge,other), counted once per event per reasonAlso reports
dynamic_instrumentation.enabledin the telemetryproductspayload.Motivation
Rate limits are decided by the probe sampler inside the V8 breakpoint condition on the main thread, capture limits are enforced in the debugger worker. To keep both hot paths cheap the counters live in a
SharedArrayBufferincremented with atomics, and the main thread drains them into telemetry metrics every 10 seconds and when the debugger stops.Additional Notes
Namespace and tag set follow the shipped .NET and Python implementations.
probe_idis intentionally not a tag.