Skip to content

test(otel-integration): cover the OTel eBPF profiler preset and validate its config against the pinned image - #987

Open
royfur wants to merge 1 commit into
masterfrom
fix/ebpf-profiler-config-validation
Open

royfur wants to merge 1 commit into
masterfrom
fix/ebpf-profiler-config-validation

Conversation

@royfur

@royfur royfur commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Closes the test gaps that let the profiler crash ship. The crash itself is fixed upstream in coralogix/opentelemetry-helm-charts#497, which landed here via #988 (opentelemetry-collector 0.136.4, chart 0.0.335) — so this PR is now test coverage only, on top of that.

Background

Profiling was broken in otel-integration 0.0.333: the chart rendered tracers: all on the profiling receiver, a key removed from the profiler in v0.156.0, so every profiler pod crash-looped with 'config.Config' has invalid keys: tracers. 0.0.333 (#984) is where the image moved to 0.156.0.

Two gaps let it through

  1. The broken path had no golden coverage. The ebpf-profiler case uses values-ebpf-profiler.yaml, which enables the legacy coralogix-ebpf-profiler sub-chart — not the opentelemetry-ebpf-profiler sub-chart the wizard emits and that users actually run.
  2. Golden diffs structurally cannot catch this. When upstream removes a config key, the rendered YAML stays byte-identical; only the binary's acceptance of it changes. The check was green while every profiler pod was crash-looping.

Changes

  • New golden case otel-ebpf-profiler built from values-otel-ebpf-profiler.yaml, i.e. the wizard configuration from the report (otlpExporter to the node agent, resourceDetection, fleetManagement without supervisor, samplesPerSecond: 20). The legacy ebpf-profiler case is left in place; it can be dropped when that sub-chart is retired.
  • New .github/scripts/check-collector-config-validity.sh, wired into the helm-golden-render job. For every golden render it extracts the collector config from each ConfigMap and, for configs that declare the profiling receiver, runs otelcol validate against them. Details:
    • It runs inside the profiler image the chart actually renders (ghcr.io/…/opentelemetry-collector-ebpf-profiler:0.156.0, read out of the golden), so the check follows every dependency bump with no extra maintenance and needs no separate binary download.
    • validate unmarshals the config and builds the pipelines — no eBPF probes, no privileges, no cluster. It is the same code path that produced the crash message in production.
    • ${env:…} references are stubbed, since validate resolves them. The config is passed in via --config=env:… rather than a bind mount, so it behaves the same in CI and on a laptop whose Docker VM doesn't share $TMPDIR.
    • Because pipelines are built for real, the eks resource detector constructs a Kubernetes client; a stub service-account token is written inside the container so it can. Nothing reads it and no request is made.
    • The script fails if no profiling config is found in the goldens, so this coverage cannot silently disappear again.

Validation

Both checks run clean locally, and the new one was verified in both directions:

  • check-helm-golden-renders.sh — clean, including the new case.
  • check-collector-config-validity.shOK otel-ebpf-profiler/…configmap_agent_yaml.
  • Re-inserting tracers: all into the golden makes it fail with exactly the production error: 'config.Config' has invalid keys: tracers.
  • shellcheck (v0.10.0, as CI pins) clean on both scripts; helmlog validate and make check-docs clean.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5072eb4543

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread otel-integration/k8s-helm/Chart.yaml
…ate its config against the pinned image

Add an `otel-ebpf-profiler` golden case built from the `opentelemetry-ebpf-profiler`
sub-chart, i.e. the configuration the wizard generates. The existing
`ebpf-profiler` case only exercises the legacy `coralogix-ebpf-profiler`
sub-chart, so the path that crash-looped in 0.0.333 had no coverage.

Add check-collector-config-validity.sh, wired into the helm-golden-render job.
For every golden render it extracts the collector config from each ConfigMap
and, for configs that declare the `profiling` receiver, runs `otelcol validate`
inside the exact profiler image the chart renders, so the check follows every
dependency bump with no extra maintenance. Golden diffs cannot catch an upstream
config-key removal on their own: the rendered YAML stays byte-identical while
the binary starts rejecting it.
@royfur
royfur force-pushed the fix/ebpf-profiler-config-validation branch from 350e795 to 1e7a02a Compare August 10, 2026 10:19
@povilasv

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1e7a02ae6e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

esac
done < <(grep -oE '\$\{env:[A-Za-z_][A-Za-z0-9_]*\}' "$config" | sed "s/\${env://; s/}//" | sort -u)

CX_COLLECTOR_CONFIG="$(cat "$config")" docker run --rm --entrypoint sh "${env_args[@]}" "$image" -c '

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid launching a shell in the collector image

When the new workflow reaches this command, Docker overrides the profiler image's entrypoint with sh, but the official collector release image is built as a minimal shell-less runtime, so the container exits with an executable-not-found error before mkdir or otelcol validate can run. This makes the required helm-golden-render job fail for every PR; run the collector binary directly and supply the service-account fixture through a mounted directory or another container rather than executing shell setup inside this image.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants