feat(kafka): add dead-consumer scenario and refinements on top of #190 - #227
Draft
nr-jonathanpdx wants to merge 6 commits into
Draft
nr-jonathanpdx wants to merge 6 commits into
nr-jonathanpdx wants to merge 6 commits into
Conversation
|
|
nr-jonathanpdx
changed the title
nr-jonathanpdx
force-pushed
the
kafka-scenario
branch
from
August 17, 2026 04:52
720cfda to
5b67833
Compare
|
This PR was marked stale due to lack of activity. It will be closed in 7 days. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
|
no stale, will be reviewed, after we upgrade the demo to V3 |
|
This PR was marked stale due to lack of activity. It will be closed in 7 days. |
nr-jonathanpdx
force-pushed
the
kafka-scenario
branch
from
September 14, 2026 21:49
5b67833 to
d4fd558
Compare
fraud-detection leaves the orders consumer group (unsubscribes and stops polling) while checkout keeps producing, so the group empties (members -> 0) and lag climbs against a zero-member group. Toggled live via the new kafkaConsumerDead flagd flag (defaults off). Also switches kafkaQueueProblems to a per-record sleep in milliseconds (5sec/3sec/1sec/off) so the throttle is tunable from the flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an NRQL alert condition (kafka_dead_consumer_group) that fires when kafka.consumer_group.members drops below threshold for the orders group, a Consumer Group Members dashboard tile faceted by group, and documents the kafkaConsumerDead scenario in newrelic/README.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a resource/host_identity processor that upserts a stable host.id and host.name so the single Docker host resolves to one INFRA-HOST entity and services (including the Kafka broker) form a "runs on" edge to it. Without this, host.name/host.id come from the collector container's ephemeral identity, so the host entity is unstable and the broker->host relationship never synthesizes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…harness)
Makes a fresh clone runnable against any account/region by editing only
.env / .env.override, with no tracked config changes:
- Drive the collector's otlphttp/newrelic endpoint from
${env:NEW_RELIC_OTLP_ENDPOINT}, passed through on the otel-collector
service; prod (US) default lives in tracked .env.
- Fix the compose build context (./ -> ../..) so the file runs from
newrelic/docker/.
- Remap only the host binding of frontend-proxy via
${ENVOY_HOST_PORT:-${ENVOY_PORT}} to avoid host port conflicts.
- Pass LOCUST_SPAWN_RATE and LOCUST_BROWSER_TRAFFIC_ENABLED through instead
of hardcoding, and add FLAGD_HOST/FLAGD_PORT to shipping.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nr-jonathanpdx
force-pushed
the
kafka-scenario
branch
from
September 15, 2026 06:12
d4fd558 to
dc0c6e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Builds on the Kafka consumer-lag observability scenario from #190 (by @crshanks), adding four improvements on top. That PR was closed; this one is opened from a fork so the work stays available and can be reviewed as a unit. Full credit for the underlying scenario is @crshanks'.
Underlying scenario (from #190)
Kafka consumer-lag observability on the New Relic side of the demo:
kafkametricsreceiver, on both the Docker and Kubernetes pathskafka.cluster.nametagging for UI facetingfraud-detectionconsumer-lag SLOThe lag-generating code itself (checkout producer burst,
fraud-detectionconsumer sleep) is pre-existing base-demo code from open-telemetry#1528 — not part of this scenario. This work only adds the New Relic observability on top.Additions in this PR
9ac8bf2kafkaConsumerDeaddead-consumer-group scenario (new flagd flag)5e0f60c30446f9720cfdaTesting
Enabling
kafkaQueueProblemsvia the flagd UI produces the expected behavior end-to-end against a New Relic account:fraud-detectionconsumer lag climbs whileaccountingstays flat, matching the scenario design. The dashboard imports and renders.Known issues to be aware of
fraud-detection's local image can be stale and silently skip the flag's sleep delay — this looks like a flagd/OpenFeature issue but isn't. Rebuild from source (--buildon compose up).derivative()withTIMESERIESon the per-partition Kafka metrics can render empty; rewriting those queries to compute the rate withoutTIMESERIESproduces the expected result.newrelic/dashboards/service_baselines.jsoncarries a stale page-levelguidplus per-widgetids; both need stripping before import or it fails with "Failed to create dashboard."--env-fileflags or it fails on unresolved env vars (720cfdamoves this to env-file-only).Merge Requirements
This is a draft opened to preserve and share the work; a few of the dashboard queries still need refinement (see the known issues above) before it's ready to merge.