Skip to content

fix(healthcheck): prevent ClickHouse broken-pipe errors in the telemetrystore healthcheck - #194

Open
julpar wants to merge 1 commit into
SigNoz:mainfrom
julpar:fix/clickhouse-healthcheck-broken-pipe
Open

fix(healthcheck): prevent ClickHouse broken-pipe errors in the telemetrystore healthcheck#194
julpar wants to merge 1 commit into
SigNoz:mainfrom
julpar:fix/clickhouse-healthcheck-broken-pipe

Conversation

@julpar

@julpar julpar commented Sep 5, 2026

Copy link
Copy Markdown

Fixes

  • Stop the ClickHouse healthcheck from writing broken-pipe errors into the ClickHouse log.

Problem

The ClickHouse healthcheck uses wget --spider. The --spider option closes the connection before it reads the response body. ClickHouse then fails to write the /ping response, and records the failure as an Error with a full C++ stack trace.

The healthcheck runs every 30 seconds, and each failure writes two error lines. This adds about 5,700 stack traces a day. The container stays healthy the whole time, because the check still exits 0, so nothing surfaces the problem. The noise hides real ClickHouse errors.

This is the cause of the long-standing report in SigNoz/signoz#5140 (open since June 2024). A user identified the healthcheck there in September 2025, and a maintainer asked for a PR. This is that PR.

Fix

Replace --spider with -O-, so wget reads the response body.

Measurements

On clickhouse-server:25.5.6, running each command 25 times and counting new StaticRequestHandler error lines:

command errors
wget --spider -q 0.0.0.0:8123/ping 22/25
wget -q -O- 0.0.0.0:8123/ping 0/25

Verified on a live single-node deployment: the error count went from a steady stream to zero, and the container stays healthy with FailingStreak=0.

Scope

Only the coolify and ecsterraform castings are affected, because they use the 0.0.0.0 host. The dockercompose and dockerswarm castings use http://localhost:8123/ping and are left unchanged.

One thing worth flagging

Please do not fix this by changing the host to localhost. In the ClickHouse image, /etc/hosts resolves localhost to ::1 first. On a host with IPv6 disabled, the connection is refused:

$ wget -q -O- http://localhost:8123/ping
wget: can't connect to remote host: Connection refused     # rc=1

$ wget -q -O- 0.0.0.0:8123/ping
Ok.                                                        # rc=0

That variant produces no broken-pipe errors only because the request never reaches ClickHouse, and it would make the healthcheck fail. It may be worth a separate look at whether the dockercompose and dockerswarm castings are exposed to this on IPv6-disabled hosts. I have not tested those flavors, so I left them alone.

Refs SigNoz/signoz#5140

…trystore healthcheck

The ClickHouse healthcheck uses `wget --spider`. The `--spider` option closes
the connection before it reads the response body. ClickHouse then fails to
write the `/ping` response. It records the failure as an `Error` with a full
stack trace.

The healthcheck runs every 30 seconds. Each failure writes two error lines.
This adds approximately 5700 stack traces each day. The container stays
healthy, because the check still exits 0, so the noise is not obvious. The
noise hides real ClickHouse errors.

This change replaces `--spider` with `-O-`, which reads the response body.

Measured on clickhouse-server 25.5.6, with 25 requests for each command:

    wget --spider -q 0.0.0.0:8123/ping    22/25 errors
    wget -q -O- 0.0.0.0:8123/ping          0/25 errors

The change applies to the coolify and ecsterraform castings. These castings
use the `0.0.0.0` host. The dockercompose and dockerswarm castings use
`http://localhost:8123/ping` and stay unchanged.

Do not change the host to `localhost` to solve this problem. In the ClickHouse
image, `localhost` resolves to `::1` first. If the host disables IPv6, the
connection is refused and the healthcheck fails.

Refs SigNoz/signoz#5140
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe-to-test Run CI on external PRs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants