Use this runbook to safely flip docktornet from dry-run to live writes against NPM, Cloudflare DNS, and Gatus. Reusable for re-cutovers; not a one-off checklist.
-
git statusis clean ondevelopment. -
just testpasses. -
just lintpasses. -
docker build -f crates/server/Dockerfile -t docktornet-server:dev .succeeds. -
docker build -f crates/agent/Dockerfile -t docktornet-agent:dev .succeeds. - At least one labelled container exists on a host you can monitor closely (a "canary" service).
The script copies the server's database out of the running docktornet-server
container, so the server must be up. On a first cutover, when it has never
run, start it once in dry-run first (DOCKTORNET_DRY_RUN=true in
deploy/server.env, then
docker compose -f deploy/main-host.compose.yaml up -d docktornet-server).
./scripts/backup_external_state.sh deploy/server.env
backup="$(ls -d docktornet-backups/[0-9]*T*Z | tail -n 1)"
ls "$backup" "$backup/npm" "$backup"/cloudflare/zones/*
git tag pre-cutover-$(date +%F)The script names each backup directory after its UTC start time,
YYYYMMDDTHHMMSSZ. It ends with backup complete: <dir>. On any failure it
exits non-zero, prints backup FAILED, and leaves what it wrote in
<stamp>.partial, which the ls -d above skips. The ls -d assumes the
default backup directory. If DOCKTORNET_BACKUP_DIR is set, use the directory
the backup complete: line names.
Verify the backup directory contains:
npm/proxy-hosts.jsoncloudflare/zones/<zone ID>/dns-records.jsonfor every managed zonedocktornet.dbgatus-config.tar.gz
Do not proceed if any file is missing.
- In
server.env: setDOCKTORNET_DRY_RUN=true. - Deploy
deploy/main-host.compose.yaml. - Tail logs for at least one
DOCKTORNET_FULL_RESYNC_INTERVAL(default 1h):
docker compose -f deploy/main-host.compose.yaml logs -f docktornet-serverExpected:
server readyand periodicsnapshot receivedlines.full resync completeexactly once per interval.dry-run: would ...lines describing the actions that would run against NPM/CF/Gatus.
Bail out if anything in the dry-run plan looks wrong (unexpected creates, deletes, or updates).
Pick one low-stakes service to flip first. The simplest way is:
-
Stop the server only (keep agents running):
docker compose -f deploy/main-host.compose.yaml stop docktornet-server
-
In
server.env:DOCKTORNET_DRY_RUN=false. -
Restart:
docker compose -f deploy/main-host.compose.yaml up -d docktornet-server
-
Verify the canary service in:
- NPM admin UI — the proxy host exists and points where intended.
- Cloudflare dashboard — the CNAME exists with the right target / proxy state.
- Gatus dashboard — the endpoint appears, the telegram alert (if enabled) fires on a forced failure.
If anything is wrong: set DRY_RUN=true, restart, then jump to §7 Rollback.
After the canary is healthy:
- Leave
DRY_RUN=false. - The next diff cycle (≤ 30 s) reconciles every other tracked service.
- Watch logs for ~10 minutes; spot-check 2–3 services across NPM/CF/Gatus.
- Push:
git push origin development git push batterforge development
- After 24 hours of clean operation, tag
v1.0.0:git tag v1.0.0 git push origin v1.0.0 git push batterforge v1.0.0
Use the backup taken in §2, the newest docktornet-backups/<stamp>/
directory:
backup=docktornet-backups/<stamp>-
Stop the server:
docker compose -f deploy/main-host.compose.yaml stop docktornet-server
-
Restore NPM hosts. Snippet (adjust to backup JSON shape):
jq -c '.[]' "$backup/npm/proxy-hosts.json" | \ while read -r host; do curl -fsS -X POST "$DOCKTORNET_NPM_URL/api/nginx/proxy-hosts" \ -H "Authorization: Bearer $NPM_TOKEN" \ -H "Content-Type: application/json" \ -d "$host" done
-
Restore Cloudflare records:
for zone_dir in "$backup"/cloudflare/zones/*/; do zone_id="$(basename "$zone_dir")" jq -c '.[]' "$zone_dir/dns-records.json" | \ while read -r rec; do curl -fsS -X POST \ "https://api.cloudflare.com/client/v4/zones/$zone_id/dns_records" \ -H "Authorization: Bearer $DOCKTORNET_CF_API_TOKEN" \ -H "Content-Type: application/json" \ -d "$rec" done done
-
Restore Gatus config:
sudo tar -xzf "$backup/gatus-config.tar.gz" -C /etc/gatus/config.d -
Wipe docktornet tracked state so it cannot try to manage resources it no longer "owns". The database is on the
./databind mount next to the compose file, owned by uid 65532:sudo rm -f deploy/data/docktornet.db deploy/data/docktornet.db-wal deploy/data/docktornet.db-shm
-
Bring docktornet back up in dry-run for a second look before reattempting:
# deploy/server.env: DOCKTORNET_DRY_RUN=true docker compose -f deploy/main-host.compose.yaml up -d