Skip to content

Latest commit

 

History

History
162 lines (125 loc) · 5.17 KB

v1-cutover.md

File metadata and controls

162 lines (125 loc) · 5.17 KB

Docktornet v1 Cutover Runbook

Use this runbook to safely flip docktornet from dry-run to live writes against NPM, Cloudflare DNS, and Gatus. Reusable for re-cutovers; not a one-off checklist.

1. Pre-flight

  • git status is clean on development.
  • just test passes.
  • just lint passes.
  • docker build -f crates/server/Dockerfile -t docktornet-server:dev . succeeds.
  • docker build -f crates/agent/Dockerfile -t docktornet-agent:dev . succeeds.
  • At least one labelled container exists on a host you can monitor closely (a "canary" service).

2. Backup

The script copies the server's database out of the running docktornet-server container, so the server must be up. On a first cutover, when it has never run, start it once in dry-run first (DOCKTORNET_DRY_RUN=true in deploy/server.env, then docker compose -f deploy/main-host.compose.yaml up -d docktornet-server).

./scripts/backup_external_state.sh deploy/server.env
backup="$(ls -d docktornet-backups/[0-9]*T*Z | tail -n 1)"
ls "$backup" "$backup/npm" "$backup"/cloudflare/zones/*
git tag pre-cutover-$(date +%F)

The script names each backup directory after its UTC start time, YYYYMMDDTHHMMSSZ. It ends with backup complete: <dir>. On any failure it exits non-zero, prints backup FAILED, and leaves what it wrote in <stamp>.partial, which the ls -d above skips. The ls -d assumes the default backup directory. If DOCKTORNET_BACKUP_DIR is set, use the directory the backup complete: line names.

Verify the backup directory contains:

  • npm/proxy-hosts.json
  • cloudflare/zones/<zone ID>/dns-records.json for every managed zone
  • docktornet.db
  • gatus-config.tar.gz

Do not proceed if any file is missing.

3. Shadow phase (dry-run)

  • In server.env: set DOCKTORNET_DRY_RUN=true.
  • Deploy deploy/main-host.compose.yaml.
  • Tail logs for at least one DOCKTORNET_FULL_RESYNC_INTERVAL (default 1h):
docker compose -f deploy/main-host.compose.yaml logs -f docktornet-server

Expected:

  • server ready and periodic snapshot received lines.
  • full resync complete exactly once per interval.
  • dry-run: would ... lines describing the actions that would run against NPM/CF/Gatus.

Bail out if anything in the dry-run plan looks wrong (unexpected creates, deletes, or updates).

4. Single-service cutover

Pick one low-stakes service to flip first. The simplest way is:

  1. Stop the server only (keep agents running):

    docker compose -f deploy/main-host.compose.yaml stop docktornet-server
  2. In server.env: DOCKTORNET_DRY_RUN=false.

  3. Restart:

    docker compose -f deploy/main-host.compose.yaml up -d docktornet-server
  4. Verify the canary service in:

    • NPM admin UI — the proxy host exists and points where intended.
    • Cloudflare dashboard — the CNAME exists with the right target / proxy state.
    • Gatus dashboard — the endpoint appears, the telegram alert (if enabled) fires on a forced failure.

If anything is wrong: set DRY_RUN=true, restart, then jump to §7 Rollback.

5. Full cutover

After the canary is healthy:

  • Leave DRY_RUN=false.
  • The next diff cycle (≤ 30 s) reconciles every other tracked service.
  • Watch logs for ~10 minutes; spot-check 2–3 services across NPM/CF/Gatus.

6. Post-cutover

  • Push:
    git push origin development
    git push batterforge development
  • After 24 hours of clean operation, tag v1.0.0:
    git tag v1.0.0
    git push origin v1.0.0
    git push batterforge v1.0.0

7. Rollback

Use the backup taken in §2, the newest docktornet-backups/<stamp>/ directory:

backup=docktornet-backups/<stamp>
  1. Stop the server:

    docker compose -f deploy/main-host.compose.yaml stop docktornet-server
  2. Restore NPM hosts. Snippet (adjust to backup JSON shape):

    jq -c '.[]' "$backup/npm/proxy-hosts.json" | \
    while read -r host; do
      curl -fsS -X POST "$DOCKTORNET_NPM_URL/api/nginx/proxy-hosts" \
        -H "Authorization: Bearer $NPM_TOKEN" \
        -H "Content-Type: application/json" \
        -d "$host"
    done
  3. Restore Cloudflare records:

    for zone_dir in "$backup"/cloudflare/zones/*/; do
      zone_id="$(basename "$zone_dir")"
      jq -c '.[]' "$zone_dir/dns-records.json" | \
      while read -r rec; do
        curl -fsS -X POST \
          "https://api.cloudflare.com/client/v4/zones/$zone_id/dns_records" \
          -H "Authorization: Bearer $DOCKTORNET_CF_API_TOKEN" \
          -H "Content-Type: application/json" \
          -d "$rec"
      done
    done
  4. Restore Gatus config:

    sudo tar -xzf "$backup/gatus-config.tar.gz" -C /etc/gatus/config.d
  5. Wipe docktornet tracked state so it cannot try to manage resources it no longer "owns". The database is on the ./data bind mount next to the compose file, owned by uid 65532:

    sudo rm -f deploy/data/docktornet.db deploy/data/docktornet.db-wal deploy/data/docktornet.db-shm
  6. Bring docktornet back up in dry-run for a second look before reattempting:

    # deploy/server.env: DOCKTORNET_DRY_RUN=true
    docker compose -f deploy/main-host.compose.yaml up -d