feat: Show health check result on failure

[zirni]

Before kamal-proxy switches traffic to a new release, it performs a health check.

If the health check fails, Kamal automatically rolls back the newly deployed container.
Once the rollback has completed, it is no longer possible to manually run a curl command
against the failed container to inspect the health check response and understand what went wrong.

To improve debugging, we execute a curl health check while the failed container still
exists and log both the HTTP status code and the response body in the Kamal deployment logs.

This has been extremely helpful when diagnosing deployment issues caused by
misconfigurations, especially in GitHub Actions, where reproducing the failed state can be difficult.

This feature can be enabled via

proxy:
  healthcheck:
    debug: true

On failure you can notice an error like this

  INFO [6bcf829d] Running docker exec kamal-proxy kamal-proxy deploy app-web-production --target="f24e8f14647c:3000" --host="somedomain.de" --deploy-timeout="30s" --drain-timeout="30s" --health-check-path="/healthz" --buffer-requests --buffer-responses --log-request-header="Cache-Control" --log-request-header="Last-Modified" --log-request-header="User-Agent" on 10.0.0.8
  INFO [2f825aa3] Running docker inspect --format '{{.NetworkSettings.Networks.kamal.IPAddress}}' f24e8f14647c | xargs -I{} curl -s -o - -w '\n%{http_code}' --max-time 5 http://{}:3000/healthz on 10.0.0.8
  INFO [2f825aa3] Finished in 0.160 seconds with exit status 0 (successful).
 ERROR Health check response:
Check <DB healthy?> failed
500
 ERROR Failed to boot web on 10.0.0.8

[Copilot]

⚠️ Not ready to approve

The new debug logging path currently suppresses useful curl error output and silently swallows exceptions, which can undermine the feature’s diagnostic value.

Pull request overview

This PR adds an opt-in “healthcheck debug” feature that captures and logs the last failed healthcheck HTTP response (status + body) during proxy-based deployments, improving post-failure diagnostics in CI/CD environments.

Changes:

• Add proxy.healthcheck.debug configuration (and documentation) to enable healthcheck-response logging on deploy failure.
• Introduce a new health_check_response command that curls the container directly to capture the response payload.
• Update boot flow to emit the captured response when the proxy deploy step fails and debug is enabled.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File summaries

File	Description
lib/kamal/cli/app/boot.rb	On proxy deploy failure, optionally captures and logs the healthcheck response when debug is enabled.
lib/kamal/commands/app/proxy.rb	Adds a pipeline command to inspect container IP and curl the healthcheck endpoint to capture status/body.
lib/kamal/configuration/proxy.rb	Adds healthcheck_debug? to expose the new proxy.healthcheck.debug setting.
lib/kamal/configuration/docs/proxy.yml	Documents the new proxy.healthcheck.debug option and default value.
test/commands/app_test.rb	Adds command-construction tests for health_check_response (default + custom path).

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 2
• Review effort level: Low

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

⚠️ Not ready to approve

The new healthcheck curl command interpolates unescaped config into a shell command and the new boot failure behavior isn’t covered by existing CLI tests.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 3
• Review effort level: Low

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

[Copilot]

⚠️ Not ready to approve

The new debug curl pipeline builds an unquoted URL argument (risking shell interpretation issues) and the docs currently imply capturing the proxy’s “last failed” healthcheck rather than a new post-failure request.

Review details

Comments suppressed due to low confidence (1)

lib/kamal/commands/app/proxy.rb:31

• The generated curl URL is not shell-quoted, so a configured healthcheck path containing shell metacharacters (e.g. ?a=1&b=2, ;, spaces) can break the command or be interpreted by the shell. Consider normalizing the path and wrapping the full URL in quotes before passing it to xargs curl.

  def health_check_response(target:)
    health_check_path = role.proxy.proxy_config.dig("healthcheck", "path") || "/up"
    app_port = role.proxy.app_port

    pipe \
      docker(:inspect, "--format", "'{{.NetworkSettings.Networks.kamal.IPAddress}}'", target),
      [ :xargs, "-I{}", :curl, "-s", "-o", "-", "-w", "'\\n%{http_code}'", "--max-time", "5",
        "http://{}:#{app_port}#{health_check_path}" ]

• Files reviewed: 5/5 changed files
• Comments generated: 2
• Review effort level: Low

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

[Copilot]

🟡 Not ready to approve

The validation schema rejects healthcheck configuration, and unescaped paths can break the diagnostic command.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (2)

lib/kamal/commands/app/proxy.rb:31

• health_check_path is configuration input and is inserted into a shell pipeline without escaping. A valid path containing a query string such as /health?quick=true&full=true causes the shell to interpret &, so curl runs in the background and the diagnostic response is lost; other metacharacters can alter the command further. Shell-escape the complete URL before adding it to the command.

        "http://{}:#{app_port}#{health_check_path}" ]

lib/kamal/cli/app/boot.rb:61

• The new failure-path wiring is not covered: the existing deploy-failure test exercises only the disabled case, while the added tests verify command construction rather than that debug: true captures and logs the response before rollback and still re-raises the deploy failure. Add a boot test for this branch so the feature's trigger and exception behavior cannot regress.

        rescue SSHKit::Command::Failed
          show_health_check_response(endpoint) if role.proxy.healthcheck_debug?
          raise

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Balanced

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

Health-check URLs are not shell-escaped, and the implementation relies on curl without ensuring it exists on supported hosts.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (2)

lib/kamal/cli/app/boot.rb:61

• The new failure-path behavior is not exercised by the CLI tests, despite test/cli/app_test.rb covering proxy deploy failures. Add coverage showing that a failed deploy invokes and logs the debug probe only when healthcheck.debug is enabled, and that probe failure does not replace the original deployment error.

        rescue SSHKit::Command::Failed
          show_health_check_response(endpoint) if role.proxy.healthcheck_debug?
          raise

lib/kamal/commands/app/proxy.rb:30

• This introduces curl as a remote-host runtime dependency, but Kamal only verifies Docker and its bootstrap can install Docker using wget; hosts with preinstalled Docker also need not have curl. On those supported hosts, debug mode cannot produce the promised response. Execute the probe through a guaranteed dependency, or explicitly validate and report/document the curl requirement.

      [ :xargs, "-I{}", :curl, "-s", "-o", "-", "-w", "'\\n%{http_code}'", "--max-time", "5",

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Balanced

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.