fix(signoz): prevent ClickHouse broken-pipe errors in the healthcheck - #11639
Closed
julpar wants to merge 1 commit into
Closed
fix(signoz): prevent ClickHouse broken-pipe errors in the healthcheck#11639julpar wants to merge 1 commit into
julpar wants to merge 1 commit into
Conversation
The ClickHouse healthcheck uses `wget --spider`. The `--spider` option closes
the connection before it reads the response body. ClickHouse then fails to
write the `/ping` response. It records the failure as an `Error` with a full
stack trace.
The healthcheck runs every 30 seconds, and each failure writes two error
lines. This adds approximately 5700 stack traces each day. The container stays
healthy, because the check still exits 0, so the noise is not obvious. The
noise hides real ClickHouse errors.
This change replaces `--spider` with `-O-`, which reads the response body.
Measured on clickhouse-server 25.5.6-alpine, the image this template pins,
with 25 requests for each command:
wget --spider -q 0.0.0.0:8123/ping 22/25 errors
wget -q -O- 0.0.0.0:8123/ping 0/25 errors
The signoz healthcheck keeps `--spider`. It uses a Go server, which closes the
connection correctly.
Do not change the host to `localhost` to solve this problem. In the ClickHouse
image, `localhost` resolves to `::1` first. If the host disables IPv6, the
connection is refused and the healthcheck fails.
Contributor
|
This PR did not pass quality checks so it will be closed. If you believe this is a mistake please let us know. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The SigNoz ClickHouse healthcheck uses
wget --spider. The--spideroption closes the connection before it reads the response body. ClickHouse then fails to write the/pingresponse, and records the failure as anErrorwith a full C++ stack trace.The healthcheck runs every 30 seconds, and each failure writes two error lines. That is about 5,700 stack traces a day. The container stays
healthythe whole time, because the check still exits 0, so nothing surfaces the problem. The noise hides real ClickHouse errors.Upstream tracking issue: SigNoz/signoz#5140 (open since June 2024).
Fix
Replace
--spiderwith-O-, sowgetreads the response body.Measurements
On
clickhouse-server:25.5.6-alpine— the image this template pins — running each command 25 times and counting newStaticRequestHandlererror lines:wget --spider -q 0.0.0.0:8123/pingwget -q -O- 0.0.0.0:8123/pingVerified on a live Coolify deployment of this template: the error stream went to zero, and the container stays
healthywithFailingStreak=0.Scope
Only the
clickhousehealthcheck changes. Thesignozhealthcheck keeps--spider— it targets a Go server, which closes the connection cleanly and produces no such errors.Only
templates/compose/signoz.yamlis touched.templates/service-templates.jsonis left alone, matching how previous changes to this template were submitted.One thing worth flagging
Please do not fix this by changing the host to
localhost. In the ClickHouse image,/etc/hostsresolveslocalhostto::1first. On a host with IPv6 disabled the connection is refused:That variant shows no broken-pipe errors only because the request never reaches ClickHouse, and it would make the healthcheck fail.
Related
The same fix for SigNoz's own Coolify casting template: SigNoz/foundry#194