[saltas888]

Superseded by #11289. The six INFP-507 PRs are consolidated there into one commit targeting develop, with the complete design, code-flow diagrams and contract-based review guide. This PR is closed to keep one active review; its branch and previous discussions are preserved.

Stacked PR 2/6 for INFP-507 (task history and activity log retention), in GitHub stack #10873 on develop. Base: #10862. Review and merge bottom-up.

1. feat(backend): add task manager retention settings [INFP-507] #10862: retention settings applied to Prefect
2. This PR: task history cleanup command, and the upgrade's task history hint
3. fix(backend): keep the Activities page fast with a year of activity log [INFP-507] #10876: Activities page queries
4. test(backend): guard the Prefect event-type list on the full stack [INFP-507] #10877: guard for Prefect's event types on the full stack
5. feat(backend): run the task manager background services with the retention settings [INFP-507] #10878: infrahub tasks background-services
6. docs: document task history and activity log retention [INFP-507] #10879: documentation

Ship with #10862 in the same release. #10862 on its own switches on Prefect's hourly deletion, which works through a large backlog slowly and gives no disk space back; this PR adds the one-off cleanup and the table rewrite.

Why

The large instances known to OpsMill hold 20 to 35 GB of task history. Prefect's own cleanup took 2 h 26 min on 25 GB in the benchmark, and Postgres does not shrink its files after deletes, so the disk only comes back after a rewrite of the tables. Design doc: Task history and Activity log retention (D9 in the numbering this stack uses, D7 since the doc renumbered its decisions on 11 Oct); spec set in dev/specs/infp-507-task-history-retention/.

Goal: delete the backlog fast with SQL that follows Prefect's own rules, rewrite each table that is mostly free space, and tell operators about it during the upgrade without slowing the upgrade.

The design doc's Design changes for implementation list asked for two changes to this PR, both done:

• 11 Oct: the one-off cleanup leaves the upgrade. The upgrade step, its renumbering, --no-task-history-cleanup and the --check text are removed, and the upgrade prints a hint instead.
• 8 Oct: the rewrite also covers the activity log's two tables, and rewrites each table only when most of its space is free.

Closes part of INFP-507.

What changed

Behavioural changes:

• infrahub tasks flush flow-runs [--rewrite] deletes, in the task manager, the finished top-level runs older than the task history retention with their task runs, states, logs and artifacts, one day of end times per transaction, and prints progress per day.
  • --rewrite then rewrites, with VACUUM FULL, each of flow_run, flow_run_state, task_run, task_run_state, log, artifact, events and event_resources whose live rows hold less than half of its disk space, whoever deleted the rows. The instance does not need to be stopped: tasks that write to a table wait while it is rewritten.
  • The summary gives the runs deleted, the size of the eight tables before and after, then one line per outcome: Rewrote …, Left … as they are, because their live rows hold at least half of their disk space, Did not rewrite …, which stayed locked by other sessions, Did not rewrite …, whose rewrite took longer than its time limit and was stopped.
  • Breaking: --days-to-keep and --batch-size are removed; the cutoff comes from the task manager's retention, set from task_manager.retention.task_history (changelog +flush-flow-runs.changed.md).
  • Against a task manager that does not provide the cleanup (older release), it prints a message and exits 0.
• infrahub upgrade deletes no task history and keeps its six steps. In its task manager step it prints Task history takes 24.8 GB, with 1,234 runs older than the retention. To get the space back, run: infrahub tasks flush flow-runs --rewrite. The count stops at 100,000 (more than 100,000 runs), so it stays fast on millions of runs; with no runs older than the retention, the step only shrinks the tables. The size and the count are read within 10 s. A task manager that answers late, answers with an error (404 from a release without the route) or answers without a size gets Could not read the task history size from the task manager (<error type>). before the command, and the upgrade goes on.
• While it waits (a cleanup runs on another task manager, the task manager lost the job, or it restarts), the command prints one line when the wait starts and when its reason changes; when it gives up after 3 hours, it names the last reason. A failed cleanup prints what it had already deleted and rewritten.
• New task-manager routes:
  • POST /infrahub/task-history/cleanup ({"rewrite": "never" | "if_freed"}) and GET /infrahub/task-history/cleanup/{id}, a background job per task manager;
  • GET /infrahub/task-history/size, the disk space of the six task history tables with their indexes (null on SQLite), and the finished top-level runs older than the retention, counted up to 100,000 (old_runs, old_runs_limit_reached).

Implementation notes:

• Same eligibility as Prefect 3.8.6's vacuum_old_flow_runs (top-level, terminal state, end_time before the cutoff, which is Prefect's own db_vacuum.retention_period), same order (runs first, then their logs and artifacts), runs selected with FOR UPDATE SKIP LOCKED, so it never waits on Prefect's own cleanup. Passes repeat until nothing is eligible, which catches subflows left without a parent.
• No DDL on Prefect's database: no tables, temporary tables, indexes or extensions. Deleted IDs travel through Python in batches under the bind-parameter limit.
• One job across task-manager replicas: a Postgres advisory lock held on a dedicated connection (a process lock on SQLite). Another replica answers 409; the CLI waits and posts again, and also waits out a task-manager restart while it follows a job. Committed days are never redone.
• A start request that reaches a task manager already running a cleanup raises that job's rewrite mode to if_freed when it asks for it, and the job decides again before it completes; it never rewrites twice.
• Free space, per table: its size on disk (main data and TOAST, without indexes) against the size of its live rows (per-column pg_column_size plus 28 bytes a row, so no value stored out of line is read). It measures space rather than counting deleted runs, because Prefect's own cleanup usually deletes the old records before the command runs.
• Up to 10,000 pages a table is read whole; above that its live size is estimated from TABLESAMPLE SYSTEM over about 10,000 random pages, which is built into Postgres. On a restored copy of Fatih's 1.10 large backup (12 GB events, 12 GB event_resources), the estimate was within 2 % of the full read and took 0.2 to 0.7 s against 4 to 6 s. A fully live table measures about 0.9 of its disk size, so it is left as it is.
• A small table usually counts as mostly free, because space is counted in whole pages, and is rewritten in milliseconds.
• The upgrade's size comes from pg_total_relation_size, which reads no table data.
• Each VACUUM FULL runs in autocommit with the limits of the manual cleanup of the event tables of OpsMill's production task manager on 26 Oct (changed on 27 Oct, from lock_timeout = 60s with 3 retries):
  • lock_timeout = 3s, retried up to 4 times 20 s apart, then listed as not rewritten. While a VACUUM FULL waits for its lock, every later query on the table queues behind it, so the wait stays short.
  • statement_timeout = 60s: a stopped VACUUM FULL leaves the table as it was, and the job lists it in a new timed_out field.
  • A plain VACUUM (ANALYZE) of each rewritten table, which blocks nothing.
  • Before the next table, a pause of ten times as long as the previous table was locked, up to 60 s, so Prefect catches up on the writes that waited.
  • The job goes on with the next table after one it could not rewrite.
• The job uses its own small connection pool with a long statement timeout, separate from Prefect's API engine.
• The request, job and size models live in prefect_server/task_history_models.py, so the CLI client imports them without loading Prefect's database layer or asyncpg (a unit test checks it).

The design doc's two suggestions are implemented: the command names each table it skipped and why, and the hint counts the old runs. The count selects the runs as the cleanup does and stops at 100,001, because an exact count would read every old run, millions on the instances this is for.

Needs approval

• Constitution deviation: the cleanup route deletes data without authentication, against Principle VI. It follows the task manager's existing trust model: reachable only on the internal network, like Prefect's own API, and only the Infrahub CLI calls it. Its input is limited to the rewrite mode; the cutoff comes from the task manager's own settings.
• New endpoint: backend/AGENTS.md lists new REST endpoints under "Ask first". GET /infrahub/task-history/size is a new read-only route on the task manager, not on the Infrahub API, so it is not in schema/openapi.json; like the other task manager routes, it has no authentication. The design doc's hint needs it, because the upgrade runs in the Infrahub server container, which reaches the task manager only over HTTP.

Known limitations

• Behind a load balancer that spreads each request across task-manager replicas, a status read can reach a replica that does not know a job that already completed; the client then starts a new job, which with --rewrite measures each table again and rewrites those still mostly free. With a Kubernetes Service and keep-alive connections, reads stay on one pod.
• Artifact collections pointing at a deleted artifact are reconciled by Prefect's own hourly service, not by this job.
• How long a rewrite of the activity log tables takes after a long activity log retention was lowered was not measured (design doc, "Cleanup step for existing instances: timings and costs").

How to review

1. backend/infrahub/prefect_server/task_history.py (job, lock, deletes, per-table rewrite, routes).
2. backend/infrahub/task_manager/flow_run/cleanup.py (client loop and size read), then cli/tasks.py and cli/upgrade.py.
3. Tests: tests/component/task_manager/test_task_history_cleanup.py runs Prefect's vacuum_old_flow_runs on one copy of a seeded database and the new cleanup on another, and compares every remaining run, task run, state, log and artifact ID (plus a concurrent run of both), and checks the per-table rewrite decision; test_task_history_routes.py; tests/integration_docker/test_task_history_cleanup.py on Postgres through the real command, including the sampled measurement and a locked table; unit tests for the client, the CLI and the upgrade's hint.

How to test

uv run pytest backend/tests/unit/task_manager/flow_run/test_cleanup.py backend/tests/unit/cli/test_tasks_flush_flow_runs.py backend/tests/unit/cli/test_upgrade_task_history.py -n 0
uv run pytest backend/tests/component/task_manager/test_task_history_cleanup.py backend/tests/component/task_manager/test_task_history_routes.py -n 0
uv run pytest backend/tests/integration_docker/test_task_history_cleanup.py -n 0

Impact & rollout

• Backward compatibility: flush flow-runs options removed (breaking, changelog). The infrahub-taskmanager flush flow-runs maintenance tool passes both options, and fails on this release without opsmill/infrahub-backup#179, which runs the command without a number of days from 1.13 on; the changelog points the tool's users to it. Deletion is irreversible; set a longer task_history before upgrading to keep older runs.
• Performance: in the benchmark, the command took about 8.5 min on 25 GB and about 2 h 20 min on 100 GB of task history, almost all of it the deletes; the upgrade itself takes no longer than before. A rewrite needs free disk for a copy of the rows a table keeps (under 0.5 GB for 25 GB of task history, 2.2 GB for 100 GB). Tasks that write to a table waited up to 7.4 s with the instance running on 25 GB, and the slowest table took about 20 s on 100 GB.
• Deployment notes: the Helm chart's upgrade hook needs no extra option. Existing instances with a lot of task history run the command once after the upgrade, the same on Compose and Helm (PR 6 documents it).

Validated locally on 27 Oct, after the change to the rewrite limits and the rebase onto develop: unit and component tests (576 passed), the integration-docker tests against an image built from the rebased stack (6 passed, including a rewrite stopped at its time limit), this PR alone without the PRs above it (169 tests passed, ruff), mypy and the generated docs.

Validated locally on 11 Oct: unit, component and integration-docker tests (the integration tests against an image built from this branch, 5 passed); the free-space sampling against a restored copy of Fatih's 1.10 large backup; quickstart Part 1 steps 5 and 6 on a dev stack with 30,000 old runs (results in #10879); /pre-ci checks (ruff, mypy, generated docs, test shards, changelog style). Earlier, on 4 to 8 Oct: the Compose upgrade guide run end to end with 120,305 old runs, a Postgres 14 and 18 probe of the advisory lock, concurrency and the lock timeout, and cubic review (three passes). Release evidence on restored 25 GB and 100 GB backups comes from opsmill/infrahub-private-tests, run separately.

Checklist

• Tests added/updated
• Changelog entry added
• External docs updated (PR 6 of this stack)
• Internal .md docs updated (PR 6 of this stack)
• I have reviewed AI generated content

🤖 Generated with Claude Code

[cubic-dev-ai]

2 issues found across 26 files

Confidence score: 4/5

• test_cleanup.py changes both the day and run count, so it doesn’t verify that a count-only increase reports progress. Add a poll with the same day and a higher deleted_runs count.
• task-manager-api.md says rewritten can be true even when no table was rewritten, including when all tables are skipped for lock timeouts. Clarify that it indicates the rewrite pass completed.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="backend/tests/unit/task_manager/flow_run/test_cleanup.py">

<violation number="1" location="backend/tests/unit/task_manager/flow_run/test_cleanup.py:84">
P2: This poll changes both the day and run count, so it does not verify that a count-only increase reports progress. Add a poll that keeps the same day while increasing `deleted_runs`, and assert that notification.</violation>
</file>

<file name="dev/specs/infp-507-task-history-retention/contracts/task-manager-api.md">

<violation number="1" location="dev/specs/infp-507-task-history-retention/contracts/task-manager-api.md:32">
P2: `rewritten` can be true even when no table was rewritten: the job sets it after the rewrite pass returns, including when every table is skipped for lock timeouts. Document it as indicating that the rewrite pass completed, and let `not_rewritten` identify skipped tables.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[codspeed]

Merging this PR will not alter performance

✅ 13 untouched benchmarks

---

_{Comparing task-history-cleanup-infp-507 (5e2d507) with task-history-retention-infp-507 (ff6a8ce)}