Skip to content

Latest commit

 

History

History
373 lines (314 loc) · 20.6 KB

cache-operations.md

File metadata and controls

373 lines (314 loc) · 20.6 KB

Cache Operations

Use this guide after configuring a public-content cache. Cache operations do not replace application authentication, deployment rollback or backups. Commands must use the same configuration, identity and filesystem permissions as the server. Inspect output privately: keys, paths and tags can reveal application structure even when response bodies are not printed.

Current Invalidation Warnings

  • Bounded purges report unfinished traversal or failed work through truncated and repeat_required. A completed traversal is not a fence against concurrent fills or content already delivered to a client.
  • Storage-bin soft purge needs room for a second encoded copy of the object. If the bin is full, the existing copy remains fresh and the operation reports unfinished work. Free space before retrying; a retry is not a hard purge.
  • Disk stale preview (dry_run=true) reads objects without deleting them, updating LRU timestamps, or scheduling an index checkpoint. Like cleanup, it requires the control token. It advances its own bounded scan cursor; concurrent traffic can still change the cache and the OS may update file access times. Preview is not an atomic inventory.

Use targeted exact/bulk invalidation for known public objects and verify subsequent responses/origin counts. For a bounded index purge, inspect the remaining scope alongside completion flags. Avoid unrestricted retry loops on a changing production cache. Changing a namespace or safely switching to a fresh root is an alternative when complete invalidation cannot be established; see recovery. Neither invalidation nor a namespace change revokes content already delivered to clients or other caches.

Choose The Right Tool

Question Tool What it proves
Would this request qualify, and which policy/key applies? cache-key Configuration-derived identity, not a live cache hit
Is a persisted filesystem object present and fresh? cache-lookup Metadata visible to a separate process; not live RAM or another process's storage-bin allocator
Can the running server fill and reuse an object? cache-warm plus status assertions Real HTTP behavior through the serving listener
What is happening in the running cache? Admin GET /_fluxheim/cache/status Live policy/activity/occupancy summary
Can I invalidate content? Protected admin purge endpoints Local-node invalidation with bounded progress/results

These CLI commands require appropriate compiled features. cache-key and cache-lookup need cache and proxy support. Configuration validation still applies; running inspection as root is not a substitute for fixing service permissions. cache-keygen generates a secret, not a cache identity; see encryption before using it.

Inspect Before Warming

For a configuration without [[vhosts]], inspection uses the root proxy/cache policy and reports scope: root, an empty vhost name and the runtime user tag root. Supply --host explicitly; it remains part of the configured cache key. No synthetic vhost is added to your configuration:

fluxheim --config examples/cache/local.toml cache-key \
  --host cache.test --path /asset.css --expect-eligible --expect-scope root

When vhosts exist, root policies are not a fallback inspection scope. Exact hosts take precedence over wildcard hosts, the longest wildcard suffix wins, and strict host routing rejects unmatched hosts. Non-strict routing uses the configured default vhost (or first vhost), just as serving does. Route matching uses exact, longest prefix, first regex, then fallback precedence and honors method restrictions. A matching route without a cache does not inherit the vhost fallback cache.

For named-vhost inspection, combine the loopback example with the explicit-vhost fragment. Run from the repository root; the generated file is local test configuration:

cat examples/cache/local.toml examples/cache/operations.toml > examples/cache/operations-local.toml
fluxheim --config examples/cache/operations-local.toml --validate-config
fluxheim --config examples/cache/operations-local.toml cache-key \
  --host cache.test --path /asset.css --expect-eligible \
  --expect-memory-tier-enabled --expect-storage-tiers 2

fluxheim --config examples/cache/operations-local.toml cache-key \
  --host cache.test --path /asset.css --method HEAD --expect-ineligible

Use --query 'version=2' separately from --path for key/lookup commands. Supply variant headers with repeated --header 'Name: value'. --host selects the configured virtual host; its name and its HTTP hostname can differ. Route selection follows the configuration rather than a CLI route override.

Preview applies configured request bypass rules, including credential, cookie and no-store bypass. It is not a simulation of authorization, rewrites, Wasm, response admission or live cache-pass state. Rewritten routes and non-proxy actions return an explicit ineligible reason rather than a guessed proxy key. The key shown is the primary key; stored response Vary fields are resolved during filesystem lookup using the supplied headers. Repeated header values are preserved. Use live HTTP tests for transformations and dynamic decisions.

The fragment explicitly enables a vhost-owned memory/filesystem cache; it does not inherit the root cache policy. Prepare its service-owned private examples/cache/state/cache directory as described in the caching guide, start the server with this combined configuration and warm the named host. Then:

fluxheim --config examples/cache/operations-local.toml cache-lookup \
  --host cache.test --path /asset.css --require-object

Lookup prints freshness, status, body length, tier, tags and header names, not the cached body. An empty lookup is expected for a memory-only server because the command cannot inspect another process's RAM. It does not fetch an origin to repair a miss. Encrypted storage needs the configured key/provider access. Use the same host/query/variant inputs as the original request.

Standalone storage-bin lookup returns an explicit unsupported error for an eligible request, even when the server is stopped. It never constructs a second allocator or takes the serving process's lease. cache-key still previews bin policies. An unsupported lookup is not an empty-cache result. Use live responses and protected status instead. The recovery guide explains the single-owner rule, reload limitations and safe layout changes.

Admin scope compatibility is unchanged: scoped purge targets are configured vhosts and routes, not a new root alias. A configured vhost literally named root remains that vhost. Root-only policies have no vhost detail in the current status API; process-wide activity totals can still include their traffic. Use named vhosts when scoped admin management is required. This CLI correction does not add an unauthenticated or in-process object inspection endpoint.

For automation, both commands expose policy expectations such as scope, vhost, route, namespace, lock, predictor, origin protection, peer fill and tier counts. Lookup additionally offers object/freshness/status/header/tag expectations. Run fluxheim cache-key --help and fluxheim cache-lookup --help for the exact flags; the backend reference explains their output and assertions. An assertion failure reports a failed check, but does not undo earlier filesystem work. Standalone filesystem inspection can rebuild a cache view and perform recovery/pruning; disk lookup is not guaranteed read-only. Do not run it against production state when a non-mutating inventory is required.

Warm Through The Server

Start the origin and Fluxheim as shown in the caching guide. For a cold cache:

fluxheim --config examples/cache/local.toml cache-warm \
  --listen 127.0.0.1:8088 --host cache.test --path /asset.css \
  --repeat 2 --expect-cache-status-sequence MISS,HIT

The first request should fetch the origin; the second should reuse the object. If it was already warm, HIT,HIT is legitimate and the cold-cache assertion will fail. Do not purge production content merely to satisfy a demonstration. For an already deployed cache, --expect-cache-status HIT checks every attempt. Use your configured --cache-status-header when it differs from x-cache-status.

cache-warm sends plaintext HTTP GET requests to a numeric socket address. It does not negotiate TLS or follow redirects. Use a private HTTP listener on the same host; do not point it at the HTTPS port. The default is the first server.listen; wildcard bind addresses become loopback. Other numeric addresses are accepted, so selecting a safe destination remains your job.

A warm input file contains one target per line:

# Host may be explicit, or supplied with --host for path-only lines.
assets.example.test /assets/app.css
assets.example.test /assets/logo.png?version=2
fluxheim --config /etc/fluxheim/fluxheim.toml cache-warm \
  --listen 127.0.0.1:8080 --input /trusted/deploy/warm-targets.txt \
  --header 'Accept-Language: en' --max-targets 256 --dry-run

Review the plan, then omit --dry-run. Inputs are UTF-8 and limited to 1 MiB; blank lines and lines beginning with # are ignored. Paths must start with /, fit 8192 bytes and contain no raw whitespace/control characters. Headers must pass the command's safe-header validation; never use warming to send credentials into a shared cache. Variants still need the correct configured/origin policy.

Option Default / behavior
--path / --input Repeat paths or read a target list; both contribute to the same target budget
--host For path entries; defaults to a configured vhost hostname if available
--header Repeat safe negotiation headers; not raw arbitrary HTTP
--timeout-secs 10, positive per-connect/read/write socket timeout, not a total job deadline
--max-targets 256, between 1 and 4096
--repeat 1, between 1 and 16; targets times repeat must not exceed 4096
--fail-fast Stop after the first failed request; otherwise report the batch failures
--dry-run Validate and print targets without network requests
--allow-status Add an accepted status; 2xx and 3xx already count as successful responses
--cache-status-header x-cache-status; choose the configured diagnostic header
--expect-cache-status Repeat to allow any listed status on each attempt
--expect-cache-status-sequence Comma-separated sequence with one value per repeat; cannot combine with the preceding option

Requests are sequential, not a load generator. A counted 200/302 is not proof of caching: use cache-status assertions and origin counts. --allow-status 404 is useful only when intentional negative caching is configured; it does not enable storage of 404 responses by itself.

Inspect And Invalidate The Running Cache

Use the protected admin listener described in configuration snapshots and administration. The commands below assume it is already bound to loopback port 9090 and use a private header file /run/secrets/fluxheim-admin-header containing Authorization: Bearer <your-control-token>. Create it through your secret management workflow with restrictive service/operator-only permissions; never commit it. The observer token cannot access cache status or mutate cache state.

Admin mutations also require audit admission. Set the following for the guide's local example, which otherwise uses level = "warn":

[logging]
level = "info"
format = "text"

Replace its existing logging table, do not append a duplicate. Keep audit events enabled in any RUST_LOG override too. A filtered, full or failed audit pipeline can return 503 with audit_unavailable before mutation. A post-mutation audit failure can report operation_may_have_completed; inspect state before retrying. Audit admission does not guarantee durable storage. See logging and the accepted audit boundary.

curl --fail-with-body --header @/run/secrets/fluxheim-admin-header \
  http://127.0.0.1:9090/_fluxheim/cache/status

curl --fail-with-body --request POST \
  --header @/run/secrets/fluxheim-admin-header \
  --header 'x-fluxheim-cache-vhost: assets' \
  --header 'x-fluxheim-cache-host: assets.example.test' \
  --header 'x-fluxheim-cache-path: /assets/app.css' \
  http://127.0.0.1:9090/_fluxheim/cache/purge

Use the actual configured vhost name (assets here), not automatically its hostname. Add x-fluxheim-cache-route for a route-owned policy. Exact purge defaults to GET; use x-fluxheim-cache-method and x-fluxheim-cache-query if needed. Exact identities include query values when the policy includes them; the path must not smuggle traversal or encoded separators. All stored variants of a matched primary identity are invalidated.

All routes below start with /_fluxheim/cache/. Query parameters are an alternative to the corresponding x-fluxheim-cache-* headers; encode query values correctly. These are POST routes, not JSON-body commands.

Endpoint Scope and parameters
purge One vhost, optional route, host, method, path, url_query/cache_query
purge-bulk Multiple path parameters, same identity scope/query for each
purge-index Whole indexed vhost or route; limit, batches, optional soft
purge-prefix Indexed path_prefix, such as /assets/; / is deliberately rejected
purge-wildcard Indexed pattern, such as /assets/*.css; not an arbitrary regular expression
purge-tag Indexed cache_tag, matching tags retained from configured origin tag headers
purge-stale Bounded scan of expired entries; limit, batches, optional dry_run
activity/reset Reset activity counters, without removing objects

Prefix, wildcard and tag purges also accept limit, batches and soft. For example, preview stale cleanup, then perform a targeted hard invalidation:

curl --fail-with-body --request POST \
  --header @/run/secrets/fluxheim-admin-header \
  'http://127.0.0.1:9090/_fluxheim/cache/purge-stale?vhost=assets&limit=500&dry_run=true'

curl --fail-with-body --request POST \
  --header @/run/secrets/fluxheim-admin-header \
  'http://127.0.0.1:9090/_fluxheim/cache/purge-prefix?vhost=assets&path_prefix=/assets/&limit=500&batches=4'

Hard purge is the default. On every tier, indexed soft=true marks an object stale while retaining its body and validators. It does not grant or extend a stale-serving window. For soft requests the effective limit is at most 32 entries per tier and one batch per request, even if larger values are supplied. Check repeat_required and use the effective limits in the response. Each affected storage-bin cache checkpoints its index once for the group, not once per object. Leave spare space for staged replacements. Contention or limited space can result in partial progress; no fallback hard deletion is attempted. The admin controller admits four authenticated soft-purge requests per rolling second across selectors (429 on excess). The HTTP admin purge endpoints share two blocking workers (503 when busy). For hard purge, see the storage-bin acknowledgement and crash behavior. Storage-bin also serializes soft checkpoints across all cache roots in the process. After staging a fresh replacement, allow at least five seconds before the next attempt, even if the write failed. A busy or cooling-down bin reports incomplete work without advancing its cursor. A tiered request may already have changed the memory tier; follow repeat_required, not just the HTTP status. Hard purge, filesystem and memory-only caches do not use the storage-bin cooldown. Soft purge is not suitable for removing disclosed/private content. Hard invalidation is not secure disk erasure either. Neither operation removes browser/CDN copies or copies on peers. A peer can refill an old object unless invalidation/version changes are coordinated.

Check response counts and actual remaining objects, not just HTTP status. limit bounds examined index entries per tier and batch, including entries that do not match the selector. Live generations share that budget; visiting an empty generation costs one unit. batches bounds attempts in this request. truncated means traversal or failed work remains; repeat_required asks for another bounded request and batches_exhausted identifies the request's cap. Counts are tier/object visits, not globally unique client resources. Soft purges count only fresh-to-stale changes, not repeated visits to stale copies.

Progress is shared per cache, selector and mode, including across successive requests and background ticks. Each index keeps at most 64 pending traversals and at most 8 KiB of selector text per traversal. Evicting a cursor restarts its scan; it does not claim completion. Each pass has an insertion-sequence boundary; new fills/replacements or failed mutations require another pass. Fresh entries and nonmatches no longer prevent later entries being examined. Concurrent purge contention reports continuation instead of waiting indefinitely.

Retained generations receive work in rotating order. When the limit cannot even visit every generation, continuation can remain set after all matches are removed; increase the limit or retire unused generations. Continuous fills, selector churn or persistent storage errors can also prevent completion. Set a finite retry policy and inspect errors instead of looping without a bound. This is not a filesystem inventory, a wall-clock deadline, a durable purge acknowledgement, or secure erasure. increase_limit_required remains false: fresh-prefix progress no longer requires increasing the scan limit.

For purge-stale, disk_scanned counts objects whose freshness was verified. disk_stale includes both objects inside a stale-serving window and fully expired objects (the latter are also counted in disk_expired). With dry_run=true, disk_would_purge reports that total and disk_purged is zero. Without dry-run, only confirmed removals count as purged. Unverifiable entries are not assumed stale: inspect disk_invalid (bad format, identity or integrity), disk_missing, and disk_unreadable (e.g. inaccessible storage). These are observations, not removals; check storage health before separate maintenance. A busy entry requests continuation rather than being counted as missing. Counts refer to visits, not unique objects across concurrent changes.

Exact/bulk responses distinguish purged and not-purged objects; a bounded slice continuation returns an explicit incomplete-operation error and may have already purged some objects. Repeat the same request and verify content.

Schedule Stale Cleanup

[cache_purger]
enabled = true
interval_secs = 300
limit = 512
batches = 1

This is a global background indexed disk-cleanup task, not request-time TTL or an origin warm-up job. Defaults are disabled, 300 seconds, 512 entries and one batch. Bounds are 1-86400 seconds, 1-100000 entries and 1-100 batches. Watch purger truncation, duration and entry counts before increasing work per tick.

The background task shares one entry budget across disk targets per batch and retains scan progress between ticks. It reports the final batch's continuation, not whether an earlier batch was partial. Target counts represent visits, which can repeat across batches. Monitor disk usage separately: continuous churn and storage failures can still prevent completion.

Verified Recipe Scope

scripts/smoke_cache_operations.py starts an actual local server with the checked-in named-vhost recipe, private control/observer tokens and loopback-only listeners. It exercises status, exact/header purge, bulk, prefix, wildcard, tag, indexed purge, fresh-only stale preview and activity reset, across memory, filesystem-only, storage-bin-only and both tiered policies. It verifies denied anonymous and observer access, bodies, origin counts and over-limit batch completion. Dedicated Rust fixtures cover bounded sparse scans, fresh prefixes, soft counts, retained generations, failed mutations, encryption and restart. Disk preview tests also compare the quiescent cache before and after inspection.

Do not remove live cache directories by hand. Preserve encrypted root/nonce state during supported operations, do not share one root between nodes, and do not assume reload clears every retained cache object. Verify invalidation through live responses and counters; use an explicit namespace/version change when deploying incompatible public representations. The encryption guide covers key/root rotation separately.