Skip to content

fix(signoz): prevent ClickHouse broken-pipe errors in the healthcheck - #11639

Closed
julpar wants to merge 1 commit into
coollabsio:mainfrom
julpar:fix/signoz-clickhouse-healthcheck-broken-pipe
Closed

fix(signoz): prevent ClickHouse broken-pipe errors in the healthcheck#11639
julpar wants to merge 1 commit into
coollabsio:mainfrom
julpar:fix/signoz-clickhouse-healthcheck-broken-pipe

Conversation

@julpar

@julpar julpar commented Sep 5, 2026

Copy link
Copy Markdown

Problem

The SigNoz ClickHouse healthcheck uses wget --spider. The --spider option closes the connection before it reads the response body. ClickHouse then fails to write the /ping response, and records the failure as an Error with a full C++ stack trace.

The healthcheck runs every 30 seconds, and each failure writes two error lines. That is about 5,700 stack traces a day. The container stays healthy the whole time, because the check still exits 0, so nothing surfaces the problem. The noise hides real ClickHouse errors.

Upstream tracking issue: SigNoz/signoz#5140 (open since June 2024).

Fix

Replace --spider with -O-, so wget reads the response body.

Measurements

On clickhouse-server:25.5.6-alpine — the image this template pins — running each command 25 times and counting new StaticRequestHandler error lines:

command errors
wget --spider -q 0.0.0.0:8123/ping 22/25
wget -q -O- 0.0.0.0:8123/ping 0/25

Verified on a live Coolify deployment of this template: the error stream went to zero, and the container stays healthy with FailingStreak=0.

Scope

Only the clickhouse healthcheck changes. The signoz healthcheck keeps --spider — it targets a Go server, which closes the connection cleanly and produces no such errors.

Only templates/compose/signoz.yaml is touched. templates/service-templates.json is left alone, matching how previous changes to this template were submitted.

One thing worth flagging

Please do not fix this by changing the host to localhost. In the ClickHouse image, /etc/hosts resolves localhost to ::1 first. On a host with IPv6 disabled the connection is refused:

$ wget -q -O- http://localhost:8123/ping
wget: can't connect to remote host: Connection refused     # rc=1

$ wget -q -O- 0.0.0.0:8123/ping
Ok.                                                        # rc=0

That variant shows no broken-pipe errors only because the request never reaches ClickHouse, and it would make the healthcheck fail.

Related

The same fix for SigNoz's own Coolify casting template: SigNoz/foundry#194

The ClickHouse healthcheck uses `wget --spider`. The `--spider` option closes
the connection before it reads the response body. ClickHouse then fails to
write the `/ping` response. It records the failure as an `Error` with a full
stack trace.

The healthcheck runs every 30 seconds, and each failure writes two error
lines. This adds approximately 5700 stack traces each day. The container stays
healthy, because the check still exits 0, so the noise is not obvious. The
noise hides real ClickHouse errors.

This change replaces `--spider` with `-O-`, which reads the response body.

Measured on clickhouse-server 25.5.6-alpine, the image this template pins,
with 25 requests for each command:

    wget --spider -q 0.0.0.0:8123/ping    22/25 errors
    wget -q -O- 0.0.0.0:8123/ping          0/25 errors

The signoz healthcheck keeps `--spider`. It uses a Go server, which closes the
connection correctly.

Do not change the host to `localhost` to solve this problem. In the ClickHouse
image, `localhost` resolves to `::1` first. If the host disables IPv6, the
connection is refused and the healthcheck fails.
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

This PR did not pass quality checks so it will be closed. If you believe this is a mistake please let us know.

@github-actions github-actions Bot closed this Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant