feat(obi): add OBI to otel-ecs-ec2 and otel-linux-standalone by NimrodAvni78 · Pull Request #969 · coralogix/telemetry-shippers

NimrodAvni78 · 2026-07-16T10:50:21Z

Description

Adds opt-in OpenTelemetry eBPF Instrumentation (OBI, v0.10.0) to the two non-k8s shippers, coupled to each shipper's deploy mechanism so it isn't a separate product — mirroring the existing otel-integration/k8s-helm OBI subchart integration. (Requested in this thread.)

otel-ecs-ec2

OBI runs as a privileged sidecar in the existing coralogix_otel_agent task (already host-net/host-pid/privileged with all eBPF host mounts), exporting to the node-local collector at 127.0.0.1:4317/4318.
Config authored in values.yaml (obi.config), rendered via make obi-config → obi/obi-config.yaml, written to the host by the launch-template user-data and bind-mounted read-only into the sidecar at OTEL_EBPF_CONFIG_PATH.
Terraform vars: enable_obi, obi_image, obi_image_version, obi_context_propagation, obi_config (full override).

otel-linux-standalone

OBI installed as an obi.service systemd unit (as root); the release binary is downloaded (arch-detected amd64/arm64) in user_data.sh.tmpl, mirroring the existing telemetrygen pattern.
Same helm-rendered config flow (make obi-config → obi/obi-config.yaml), injected into user-data (base64+gzip).
Terraform vars: enable_obi, obi_version, obi_context_propagation, obi_config_path.

Shared

Full-parity OBI config with k8s (ebpf buffer sizes, payload_extraction incl. GenAI, redis_db_cache, prometheus_export, gen_ai attribute select), dropping k8s-only pieces (Kubernetes metadata, k8s cache, k8s owner-name network filters) and using host-process discovery (open_ports/exe_path).
Disabled by default; new obi/ subfolder + README per target.
Standalone chart versions bumped together (linux/windows/macos → 0.0.46) to satisfy the standalone version-check; ecs chart → 0.0.41.

How Has This Been Tested?

Static validation (all passing):

helm lint + helm template (default and --set obi.enabled=true) for both charts.
make obi-config renders a valid OBI config for both targets; default make otel-config output is unaffected (OBI ConfigMap is gated behind obi.enabled).
tofu validate for both terraform modules.
The linux user_data template was rendered with enable_obi=true and passes bash -n.
OBI config content is copied verbatim from the k8s v0.10.0 rendered config (same OBI version), with only the verified discovery-schema swap.

Live smoke test (deploy each target with enable_obi=true and confirm OBI telemetry reaches Coralogix) is pending and will be run before marking ready for review.

Checklist:

I have updated the relevant Helm chart(s) version(s)
I have updated the relevant component changelog(s)
This change does not affect any particular component (e.g. it's CI or docs change)

🤖 Generated with Claude Code

Adds opt-in OpenTelemetry eBPF Instrumentation (OBI, v0.10.0) to the two non-k8s shippers, coupled to each shipper's deploy mechanism (matching the existing k8s-helm subchart integration). otel-ecs-ec2: - OBI runs as a privileged sidecar in the existing collector task (already host-net/host-pid/privileged with all eBPF host mounts), exporting to the node-local collector at 127.0.0.1:4317/4318. - Config authored in values.yaml (obi.config) and rendered via `make obi-config` into obi/obi-config.yaml; written to the host by the launch-template user_data and bind-mounted read-only into the sidecar. - Terraform vars: enable_obi, obi_image, obi_image_version, obi_context_propagation, obi_config (full override). otel-linux-standalone: - OBI installed as an obi.service systemd unit (run as root); the release binary is downloaded (arch-detected) in user_data, mirroring the telemetrygen pattern. - Same helm-rendered config flow (make obi-config -> obi/obi-config.yaml), injected into user_data (base64+gzip). - Terraform vars: enable_obi, obi_version, obi_context_propagation, obi_config_path. Shared: - Full-parity OBI config with k8s (ebpf buffer sizes, payload_extraction incl GenAI, redis_db_cache, prometheus_export, gen_ai attribute select), dropping k8s-only bits and using host-process discovery (open_ports/exe_path). - Disabled by default; new obi/ subfolder + README per target. - Standalone chart versions bumped together (linux/windows/macos 0.0.46) to satisfy the version-check; ecs chart 0.0.41. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

NimrodAvni78 · 2026-07-16T11:37:54Z

✅ Live smoke test passed (both targets, Coralogix staging / `anemia-new`)

Deployed each target to AWS with enable_obi=true, confirmed OBI telemetry reaches Coralogix, then tore everything down (0 resources left).

otel-linux-standalone

obi.service installed via the release binary + systemd unit (as root), came up active.
OBI loaded eBPF probes and discovered/instrumented host processes (incl. a test HTTP server, type=python).
Collector exporter counters: otelcol_exporter_sent_spans_total{exporter="coralogix"} = 109, send_failed_spans_total = 0.

otel-ecs-ec2

OBI ran as the privileged sidecar in the collector task; discovered/instrumented all host processes — telemetry stamped source="obi", telemetry_distro_name="opentelemetry-ebpf-instrumentation" (incl. cdot collector, ECS agent, telemetrygen, containerd, dockerd, sshd, …).
Collector exported 24,914 spans to staging with 0 failures.

Notes / findings

(pre-existing, not OBI) On otel-linux-standalone, the chart's full rendered collector config is incompatible with the pinned upstream otelcol-contrib v0.137.0 .deb: it uses the Coralogix-custom systemd receiver (absent upstream) plus coralogix.domain_settings.keepalive.enabled and spanmetrics/db.add_resource_attributes (rejected by that collector version). The base make deploy (without OBI) hits the same crash. The OBI smoke test used a minimal collector config to work around it. Worth a separate look at the deb/chart version alignment.
(minor, ECS) OBI logs a non-fatal warning creating OTEL namespace in bpffs failed … mkdir /sys/fs/bpf/otel: no such file or directory, disabling pinned-map extras (log enricher, profile correlation). Core instrumentation is unaffected. Possible follow-up: adjust the bpf host mount / pre-create the dir.
CI: the Check Documentation formatting and links failure is pre-existing external-link flakiness (opentelemetry.io/kubernetes.io timeouts) in files this PR does not touch; no errors in the added docs.

…ation The "Check Documentation formatting and links" job validates every .md file in the repo on each PR. It was failing on transient timeouts / 301 restructures of opentelemetry.io and kubernetes.io links in pre-existing files (root README.md, otel-integration/*) unrelated to this PR. Ignore those two domains in the mdox validator, matching the existing github.com "rate-limited" ignore rationale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Adds opt-in OpenTelemetry eBPF Instrumentation (OBI) integration to the non-Kubernetes shippers (ECS EC2 + Linux standalone), including config rendering via Helm and deployment wiring via Terraform/user-data, while bumping standalone chart versions in lockstep.

Changes:

Add OBI configuration (values + Helm-gated ConfigMap) and generated obi/obi-config.yaml outputs for otel-ecs-ec2 and otel-linux-standalone.
Wire OBI deployment toggles into Terraform and bootstrapping (ECS sidecar; Linux systemd unit + binary download).
Bump chart versions + changelogs (including macOS/Windows standalone alignment).

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
otel-windows-standalone/Chart.yaml	Bump chart version to align standalone releases.
otel-windows-standalone/CHANGELOG.md	Changelog entry for alignment bump.
otel-macos-standalone/Chart.yaml	Bump chart version to align standalone releases.
otel-macos-standalone/CHANGELOG.md	Changelog entry for alignment bump.
otel-linux-standalone/values.yaml	Add OBI config values (disabled by default).
otel-linux-standalone/terraform/variables.tf	Add Terraform variables controlling OBI install/runtime.
otel-linux-standalone/terraform/templates/user_data.sh.tmpl	Install OBI binary + configure/run `obi.service` when enabled.
otel-linux-standalone/terraform/main.tf	Pass rendered OBI config into user-data when enabled.
otel-linux-standalone/templates/obi-configmap.yaml	Helm-gated ConfigMap to render/extract OBI config via `make obi-config`.
otel-linux-standalone/README.md	Document OBI option and `make obi-config`.
otel-linux-standalone/obi/README.md	Detailed Linux-standalone OBI usage/config docs.
otel-linux-standalone/obi/obi-config.yaml	Generated default OBI config artifact.
otel-linux-standalone/Makefile	Add `obi-config` target and wire Terraform vars for OBI.
otel-linux-standalone/Chart.yaml	Bump chart version.
otel-linux-standalone/CHANGELOG.md	Changelog entry for OBI feature.
otel-ecs-ec2/values.yaml	Add OBI config values (disabled by default).
otel-ecs-ec2/terraform/variables.tf	Add Terraform variables controlling OBI sidecar behavior.
otel-ecs-ec2/terraform/README.md	Document OBI usage for ECS EC2 deployment.
otel-ecs-ec2/terraform/Makefile	Add example TF vars for enabling OBI and setting image tag.
otel-ecs-ec2/terraform/main.tf	Add OBI host config write + ECS sidecar container + log group when enabled.
otel-ecs-ec2/templates/obi-configmap.yaml	Helm-gated ConfigMap to render/extract OBI config via `make obi-config`.
otel-ecs-ec2/obi/README.md	Detailed ECS EC2 OBI usage/config docs.
otel-ecs-ec2/obi/obi-config.yaml	Generated default OBI config artifact.
otel-ecs-ec2/Makefile	Add `obi-config` target and include it in default make flow.
otel-ecs-ec2/Chart.yaml	Bump chart version.
otel-ecs-ec2/CHANGELOG.md	Changelog entry for OBI feature.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Reformat the Markdown tables in otel-ecs-ec2/obi/README.md and otel-linux-standalone/obi/README.md to mdox's canonical style so the 'Check Documentation formatting and links' job passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

povilasv · 2026-07-16T12:35:56Z

+    app.kubernetes.io/part-of: obi
+    app.kubernetes.io/component: config
+data:
+  obi-config.yml: |


Why is this needed? Lets not add templates here they should be in helm-charts

im not sure how we usually template out this in linux and ecs?
should we use helm even if its not k8s? or terraform / something else

povilasv · 2026-07-16T12:36:45Z

+`make otel-config` / `make manifest` renders are unaffected.
+*/ -}}
+{{- if .Values.obi.enabled }}
+apiVersion: v1


this should be in opentelemetry-helm-charts or in your subchart?

ill check but this just renders the config map even though its not k8s.
how should these configs be templated?

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 75285e25ed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-16T12:37:04Z

+  default     = ""
+
+  validation {
+    condition     = !var.enable_obi || var.obi_config_path != ""


Avoid Terraform 1.9-only variable validation

For users running the documented minimum Terraform range (required_version >= 1.5.7 in this module), this cross-variable validation is rejected before planning: Terraform only added references to other variables in validation conditions in 1.9 (HashiCorp notes this as new in 1.9). That means Linux standalone deploys fail even when OBI is disabled on Terraform 1.5.7–1.8; either raise the required version to >= 1.9 or move this check to a resource precondition/local validation pattern.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-16T12:37:04Z

    kernel.perf_event_paranoid=1
    SYSCTL
    sysctl -p /etc/sysctl.d/99-ebpf.conf
+    ${var.enable_obi ? local.obi_user_data : ""}


Avoid launch-only writes for OBI config

When enable_obi (or obi_config) is changed on an already-applied ECS cluster, this launch-template user data is the only path that writes /etc/obi/obi-config.yml. Existing ASG instances do not rerun user data just because the launch template changes, and this ASG has no instance refresh, while the ECS daemon service can roll immediately to the task definition that bind-mounts /etc/obi; those hosts then start OBI with a missing or stale config until instances are manually replaced.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-16T12:37:04Z

+  obi_container = {
+    name       = local.obi_name
+    image      = "${var.obi_image}:${var.obi_image_version}"
+    essential  = false


Add a restart policy for the OBI sidecar

If the OBI sidecar exits after the task starts, marking it non-essential keeps the collector task alive but does not restart OBI by itself; AWS documents that ECS container restart policies are not enabled by default and must be set explicitly (ECS restart policy docs). In any environment where OBI hits a transient eBPF/runtime failure, OBI telemetry stays off until the whole task is redeployed, so add a restartPolicy for this sidecar if it should recover independently.

Useful? React with 👍 / 👎.

…s-linux-standalone # Conflicts: # otel-ecs-ec2/CHANGELOG.md # otel-ecs-ec2/Chart.yaml # otel-linux-standalone/CHANGELOG.md # otel-linux-standalone/Chart.yaml # otel-macos-standalone/CHANGELOG.md # otel-macos-standalone/Chart.yaml # otel-windows-standalone/CHANGELOG.md # otel-windows-standalone/Chart.yaml

…ntation subchart Per review (Povilas): render the OBI config via the shared opentelemetry-ebpf-instrumentation subchart — the same chart the k8s-helm integration uses — instead of a bespoke values block + custom ConfigMap template, so the config is produced through Helm exactly like the collector config and the onboarding UI/backend can render it the same way. - Add opentelemetry-ebpf-instrumentation subchart dependency (0.1.18, from coralogix-charts-virtual so CI's helm dependency build resolves it) to both otel-ecs-ec2 and otel-linux-standalone, gated on opentelemetry-ebpf-instrumentation.enabled. - Replace hand-authored obi.config values with host overrides under the subchart alias (open_ports/exe_path discovery, 127.0.0.1 export endpoints, attributes.kubernetes.enable=false); ebpf/payload_extraction/redis_db_cache/ prometheus_export/health_check now come from the subchart defaults. - make obi-config extracts the subchart ConfigMap and strips the k8s network filter (cannot be removed via values). - Remove custom templates/obi-configmap.yaml; update READMEs and CHANGELOGs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

NimrodAvni78 requested a review from Copilot July 16, 2026 12:03

Copilot started reviewing on behalf of NimrodAvni78 July 16, 2026 12:04 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread otel-linux-standalone/terraform/variables.tf

NimrodAvni78 and others added 2 commits July 16, 2026 15:07

Potential fix for pull request finding

67d5a94

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

NimrodAvni78 marked this pull request as ready for review July 16, 2026 12:29

NimrodAvni78 requested review from a team, ShauliSolomovich, avivgold098 and oded-dd as code owners July 16, 2026 12:29

povilasv reviewed Jul 16, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 16, 2026

View reviewed changes

NimrodAvni78 and others added 2 commits August 12, 2026 12:28

Navigation Menu

feat(obi): add OBI to otel-ecs-ec2 and otel-linux-standalone - #969

feat(obi): add OBI to otel-ecs-ec2 and otel-linux-standalone#969
NimrodAvni78 wants to merge 6 commits into
masterfrom
nimrodavni78/obi-ecs-linux-standalone

NimrodAvni78 commented Jul 16, 2026

Uh oh!

NimrodAvni78 commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

povilasv Jul 16, 2026

Uh oh!

NimrodAvni78 Jul 16, 2026

Uh oh!

povilasv Jul 16, 2026

Uh oh!

NimrodAvni78 Jul 16, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

NimrodAvni78 commented Jul 16, 2026

Description

otel-ecs-ec2

otel-linux-standalone

Shared

How Has This Been Tested?

Checklist:

Uh oh!

NimrodAvni78 commented Jul 16, 2026

✅ Live smoke test passed (both targets, Coralogix staging / anemia-new)

otel-linux-standalone

otel-ecs-ec2

Notes / findings

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

povilasv Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

NimrodAvni78 Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

povilasv Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

NimrodAvni78 Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

✅ Live smoke test passed (both targets, Coralogix staging / `anemia-new`)