fix: skip backend pod migrations when migrationJob is enabled

[sdolidze]

Closes: https://linear.app/lightdash/issue/PROD-8562
Closes #134

Summary

When migrationJob.enabled=true, the backend Deployment pods still ran DB migrations on startup, in addition to the dedicated pre-upgrade migration Job. On a helm upgrade the Job and every rolling backend pod then ran knex migrate:latest against the same database concurrently, contending on the knex migration lock; a migrator killed mid-run left the lock held, blocking every subsequent startup.

Affects only deployments that opt into migrationJob.enabled=true (default is false). The default path — and the workers — are unchanged.

Root cause

The chart never told backend pods to skip migrating when the Job was enabled. The backend container command rendered unconditionally, and image.command is unset by default:

# templates/backendDeployment.yaml (before)
command: {{ .Values.image.command }}   # empty -> falls through to the image entrypoint

The image entrypoint (docker/prod-entrypoint.sh in lightdash/lightdash) migrates before starting the server (pnpm -F backend migrate-production then exec node dist/index.js). So enabling migrationJob added a migrator (the Job) without removing the per-pod one — 1 Job + N replicas all migrating at once. The worker templates were never affected because they already set their own non-migrating command. The pre-upgrade Job was introduced in #121, where this exact gap was raised and deferred.

Fix

When migrationJob.enabled and the operator hasn't set their own image.command, give the backend container an explicit command that starts the server without migrating, so the Job is the sole migrator. dumb-init is preserved as PID 1 (the image ENTRYPOINT wraps it) and node dist/index.js is the image's default CMD — i.e. the normal boot minus the migrate step.

• templates/backendDeployment.yaml — gate the backend command: user image.command wins if set; else when migrationJob.enabled render ["dumb-init", "--", "node", "dist/index.js"]; else render nothing (image entrypoint migrates, exactly as today — preserving the kubectl-update-without-helm path Run database migrations in pre-install,pre-upgrade hook #121 wanted to keep).
• values.yaml — document the migrationJob.enabled behavioural coupling.
• Chart.yaml / README.md — bump chart version 2.9.0 -> 2.9.1 (required by ct lint).

Out of scope: a SKIP_DB_MIGRATIONS env toggle in prod-entrypoint.sh (lives in lightdash/lightdash); the /api/v1/health connection-pool pressure during upgrades (upstream).

Before / After

Rendered templates/backendDeployment.yaml with migrationJob.enabled=true:

Before:  command:                                              # empty -> entrypoint migrates -> races the Job
After:   command: ["dumb-init", "--", "node", "dist/index.js"] # server only; Job is the sole migrator

migrationJob.enabled=false (default) — unchanged in both:

(no command line rendered) -> image entrypoint migrates, as before

Test plan

• helm lint . — 0 charts failed.
• helm template … --set migrationJob.enabled=true → backend renders the non-migrating command; the -migrate Job still renders.
• helm template … (default) → no backend command: line (image entrypoint migrates) — unchanged from before the fix.
• User override: image.command set → user value wins, with and without migrationJob.enabled.
• Regression: worker / nats deployment command lines unchanged across both modes.
• Full render parses as valid YAML in both modes (19 docs enabled / 15 default).
• CI: ct lint --all + ct install on kind (runs on the PR).

🤖 Generated with Claude Code

[linear-code]

PROD-8562