fix(grafana): point Kupo proxy panels at green multi-network proxy - #75
Merged
Conversation
The Request Latency, Request Rate and HTTP Error Rate panels in the Kupo/Dolos dashboard filtered by pod=~".*$network.*", which only ever matched the idle per-network proxies (dolos-kupo-proxy-demeter-<net>) and never the green multi-network proxy (dolos-kupo-proxy) that actually serves all Demeter Kupo traffic, leaving the panels flat. The green proxy exposes network via different labels per metric: - kupo_proxy_http_request_duration_seconds_bucket -> network label (cardano-mainnet/preprod/preview) - kupo_proxy_http_total_request -> exported_instance label (dolos-cardano-<net>.dolos.svc.cluster.local:11442) Filter the latency panels by network and the rate/error panels by exported_instance so they reflect real green-proxy traffic. Signed-off-by: Tripura Repalle <tripura@blinklabs.io>
Add a Grafana alert to the Kupo Dolos group that fires when the Kupo (Dolos) proxy returns a sustained rate of unauthenticated 401s (consumer=".") above 0.1 req/s for 15m. Kupo/Dolos themselves are healthy in this case; the pattern indicates newly issued or rotated Demeter credentials are not reaching the proxy because the demeter-fabric daemon has silently lost its Kafka/Redpanda consumer connection (readiness stays green while it stops syncing creds). There is no direct fabric Kafka metric exported, so this alerts on the customer-facing symptom, with the fabric root cause and remediation in the annotation. Signed-off-by: Tripura Repalle <tripura@blinklabs.io>
wolf31o2
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Request Latency, Request Rate and HTTP Error Rate panels in the Kupo/Dolos dashboard filtered by pod=~".$network.", which only ever matched the idle per-network proxies (dolos-kupo-proxy-demeter-) and never the green multi-network proxy (dolos-kupo-proxy) that actually serves all Demeter Kupo traffic, leaving the panels flat.
The green proxy exposes network via different labels per metric:
Filter the latency panels by network and the rate/error panels by exported_instance so they reflect real green-proxy traffic.