Skip to content

fix(service): prevent signoz broken-pipe errors - #11640

Merged
peaklabs-dev merged 1 commit into
coollabsio:feat/servicesfrom
julpar:fix/signoz-clickhouse-healthcheck
Sep 8, 2026
Merged

fix(service): prevent signoz broken-pipe errors#11640
peaklabs-dev merged 1 commit into
coollabsio:feat/servicesfrom
julpar:fix/signoz-clickhouse-healthcheck

Conversation

@julpar

@julpar julpar commented Sep 5, 2026

Copy link
Copy Markdown

Changes

The SigNoz ClickHouse healthcheck used wget --spider. That closes the connection before reading the body, so ClickHouse fails to write the /ping response and logs an Error with a full stack trace. At a 30s interval that is ~5,700 stack traces a day. The container stays healthy throughout (the check still exits 0), so nothing surfaces it. The noise buries real ClickHouse errors.

This switches it to wget -q -O-, which reads the body. Only the clickhouse healthcheck changes; the signoz one keeps --spider, as it targets a Go server that closes cleanly. templates/service-templates.json is deliberately untouched, matching earlier changes to this template.

Issues

Category

  • Fixing or updating existing one click service

AI Assistance

  • AI was used (please describe below)

If AI was used:

  • Tools used: Claude Code
  • How extensively: investigation, measurements, the one-line change, and this description. Results were verified against a running instance.

Testing

Verified on a live Coolify instance running this template, on clickhouse-server:25.5.6-alpine, the image it pins. That is a real deployment, not a local dev instance.

Each command ran 25 times in the container, counting new StaticRequestHandler errors:

wget --spider -q 0.0.0.0:8123/ping    22/25 errors
wget -q -O- 0.0.0.0:8123/ping          0/25 errors

After the change the errors stopped entirely.

One caution: do not fix this by switching the host to localhost. In the ClickHouse image it resolves to ::1 first, so with IPv6 disabled the connection is refused (wget exits 1). That shows no errors only because the request never reaches ClickHouse, and would make the check fail.

Contributor Agreement

Important

  • I have read and understood the contributor guidelines. If I have failed to follow any guideline, I understand that this PR may be closed without review.
  • I have searched existing issues and pull requests (including closed ones) to ensure this isn't a duplicate.
  • I have tested all the changes thoroughly with a local development instance of Coolify and I am confident that they will work as expected when a maintainer tests them.

The ClickHouse healthcheck used `wget --spider`, which closes the connection
before it reads the body. ClickHouse then fails to write the /ping response
and logs an Error with a full stack trace, roughly every 30 seconds. Using
`-O-` reads the body instead.

Measured on clickhouse-server 25.5.6-alpine, 25 requests each:
  --spider  22/25 errors
  -O-        0/25 errors
@julpar julpar closed this Sep 5, 2026
@julpar julpar reopened this Sep 5, 2026
@peaklabs-dev peaklabs-dev changed the title fix(signoz): prevent ClickHouse broken-pipe errors in the healthcheck fix(service): prevent signoz broken-pipe errors Sep 8, 2026
@peaklabs-dev
peaklabs-dev changed the base branch from main to feat/services September 8, 2026 17:22

@peaklabs-dev peaklabs-dev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your PR!

@peaklabs-dev
peaklabs-dev merged commit f2e0687 into coollabsio:feat/services Sep 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants