Agent Health issue for OpenMetrics metrics dropped by configured limit - #24819
Agent Health issue for OpenMetrics metrics dropped by configured limit#24819nubtron wants to merge 21 commits into
Conversation
🎉 All green!🧪 All tests passed 🔄 Datadog auto-retried 4 jobs - 4 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: 7ea240d | Docs | View more details | Give us feedback! |
nubtron
changed the title
The Fleet UI renders remediation step text as plain text, so the backticks around config option names were shown literally to customers. Match the plain-text convention used by other Agent Health producers.
The MetricLimitIssueReporter carried a boolean `legacy` flag whose only effect was selecting between the v1 (`metrics` / `ignore_metrics`) and v2 (`metrics` / `exclude_metrics`) config option names in remediation text. Pass the actual option string instead so the reporter no longer knows about OpenMetrics flavors and the call sites document themselves.
The Fleet UI renders Agent Health remediation text as plain text, so the
dotted form 'debug_metrics.metric_contexts: true' reads as a single literal
key. The check parses instance.get('debug_metrics', {}).get('metric_contexts'),
so a dotted top-level key would never match and no telemetry would be emitted.
Describe the nesting in prose ('set metric_contexts to true under the
debug_metrics section') and spell out both emitted metric names in full
instead of the orphaned '.limit'. Add assertions locking in the corrected
wording.
evalya-impact-summaryevalya impact analysis |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1584239ed8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if self.metric_limiter: | ||
| try: | ||
| reached_limit = self.metric_limiter.reached_limit | ||
| observed_count = self.metric_limiter.count | ||
| limit = self.metric_limiter.limit | ||
| self._on_metric_limit_state(reached_limit, observed_count, limit) |
There was a problem hiding this comment.
When process_isolation is enabled, run_with_isolation creates a child check and its run() already reports the over-limit issue through the redirected datadog_agent. After the child exits, this parent-side block reads the untouched parent limiter (reached_limit=False, count=0) and invokes the same deterministic callback, immediately resolving the issue that the child just reported. Limit handling should occur only in the process that performed the scrape, or the child's limiter state must be propagated back.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Looks legit will look into it!
| def _on_metric_limit_state(self, reached_limit: bool, observed_count: int, limit: int) -> None: | ||
| self.metric_limit_issue_reporter.handle( | ||
| self, | ||
| self.instance.get('openmetrics_endpoint'), |
There was a problem hiding this comment.
For V2 integrations that synthesize scraper_configs from integration-specific options, this lookup does not identify the endpoint that was scraped. For example, CiliumCheckV2 accepts agent_endpoint/operator_endpoint and only places openmetrics_endpoint in generated scraper configs; if a user enables max_returned_metrics, self.instance.get('openmetrics_endpoint') remains None, causing MetricLimitIssueReporter.handle() to return without reporting any drops. The callback needs the actual configured scraper endpoint or endpoints rather than the raw instance field.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same here, looks like a valid comment.
…oints The V2 metric-limit callback read openmetrics_endpoint from the raw instance, which stays absent for integrations such as Cilium that synthesize scraper configs from agent_endpoint/operator_endpoint, so drops were never reported. Pass the actual configured scraper endpoint keys from self.scrapers instead, and have V1 pass its one effective prometheus_url. MetricLimitIssueReporter.handle now accepts a collection of endpoints and normalizes it to a deterministic, de-duplicated, sorted tuple, discarding empty values. One aggregate issue is reported for the whole run since the limiter state cannot attribute drops to a single scraper. The machine-readable extra['endpoints'] list and the issue identity reflect the endpoint set; the existing single-endpoint public issue id is preserved, and multi-endpoint identities hash the structured ordered collection.
Validation ReportAll 21 validations passed. Show details
|
1 participant
What does this PR do?
OpenMetrics stops submitting new metric contexts after it reaches
max_returned_metrics. The check logs a warning, but the resulting gaps can look intermittent and are easy to miss.This PR surfaces that condition in Agent Health. When an OpenMetrics v1 or v2 endpoint exceeds its effective limit, the check reports:
OpenMetrics Metrics Dropped By Configured Limitopenmetrics_metrics_dropped_by_configured_limitRemediation shown to customers
The issue carries the following remediation (from
metric_limit_issue._remediation):Summary: Reduce what this endpoint sends to Datadog, or raise this instance's metric limit after checking the cost.
metrics / ignore_metrics(OpenMetrics v1) ormetrics / exclude_metrics(OpenMetrics v2) on this instance to stop collecting series you do not query, alert on, or keep.max_returned_metricson this instance to a value above the observed count.debug_metrics.metric_contexts: trueon the instance to publishdatadog.agent.metrics.contexts.totaland.limit, and confirm the total stays below the limit at peak. Consider a monitor at 80% of the limit.Each endpoint gets a stable issue identity derived from the host, check name, endpoint, and namespace. The issue is refreshed while drops continue, resolved on the first clean collection, and reused if the condition later returns.
Severity reflects the fraction of contexts dropped:
The Agent Health bridge is best-effort: reporting failures do not fail the check or interfere with the limiter's existing cleanup. The generic
AgentCheckhook remains a no-op, so only OpenMetrics v1 and v2 opt into this issue.The PR also adds the missing
issue_typeargument toAgentCheck.report_issue. The Agent payload, persistence, and forwarding paths already support this field, so no rtloader or Agent ABI change is needed.Why?
When OpenMetrics exceeds
max_returned_metrics, some metrics are silently discarded after the warning is emitted. Because exporter ordering can change between scrapes, customers may see different metrics disappear over time instead of one obvious hard cutoff.Agent Health gives this failure mode a durable, endpoint-specific state with actionable remediation, while preserving the existing check lifecycle and limiter behavior.
Review checklist
qa/skip-qais applied