Fluxheim 1.9 adds GET /_fluxheim/operational-status on the existing admin
listener. It gives a CLI or future UI one bounded view of process capabilities,
configuration counts, recovery state, recent admin changes and telemetry health.
It does not enable a listener, provide a GUI, or implement full server management.
Use the existing admin and snapshot configuration. Keep the admin listener on loopback; tunnel to it or use a properly secured TLS management gateway. Never expose a plaintext admin bearer token to the network. The configured client-certificate policy applies as for other admin routes. The new route always requires a token, even when health is public or the Unix operations socket permits other tokenless local reads.
An API-only example client needs Python's standard library, not server-file access:
# With an SSH tunnel to the existing admin port, for example 19090:
python3 scripts/operational_status_client.py http://127.0.0.1:19090The client prompts for the token without echoing it or putting it on the command line. It verifies HTTPS certificates, refuses redirects, limits the response to 16 KiB, and allows plaintext only to numeric loopback addresses. Socket operations have a five-second timeout. An HTTPS management gateway may require additional client-certificate support beyond this small example client.
Give monitoring consumers a separate, status-only credential:
[admin]
enabled = true
listen = "127.0.0.1:9090"
token_file = "/etc/fluxheim/secrets/admin-token"
operational_token_file = "/etc/fluxheim/secrets/observer-token"
snapshot_store = "/var/lib/fluxheim/snapshots"operational_token_file is optional. It authorizes only GET on this exact
endpoint, not legacy status, snapshot, recovery, reload, cache or load-balancer
routes. Both listener and Unix socket enforce this restriction. Existing tokenless
local reads on the Unix socket remain governed by its existing configuration.
The full control-plane token still works here, but do not distribute it to
monitoring consumers. Client-certificate policy and HTTP authentication throttling
also apply to observer requests.
An observer success does not reset failed-token attempts. Invalid credentials
share one bounded per-source budget across all protected routes, since a control
token can also authenticate on the observer route. Only a successful control-token
request clears that source's failures. A valid observer token used outside its
scope returns 403 forbidden without clearing or incrementing this budget.
Source lockout still blocks both roles until expiry. The existing global
invalid-attempt lockout continues to permit valid credentials for incident
response; it is not a pre-authentication CPU-rate limit.
Create an independent random token, not a copy of the admin token. The server
rejects identical credentials (including after whitespace trimming), empty or
oversized credentials and unsafe secret paths. On Unix the observer file must be
root- or service-owned, readable by the service, with no group/other permissions
(normally 0600); parent directories must be trusted. Windows uses the existing
confidential-file ACL policy. Symlinks and non-regular files are rejected. Secret
contents are read with a bound, zeroized after loading and retained only as a MAC.
Changing the file or configured path requires restarting the admin service;
reload cannot rotate credentials. Full role management remains future work.
The OpenAPI 3.1.0 description,
JSON Schema 2020-12 response,
response fixture and
error fixtures are
versioned together. The description uses the
OpenAPI 3.1 schema model.
There are no query parameters, pagination, filtering, history queries or request
bodies for this operation. A query is rejected rather than silently ignored.
Responses have Content-Type: application/json and Cache-Control: no-store.
| Field | Meaning |
|---|---|
schema_version |
Contract major version, currently 1; independent of the Fluxheim release version |
version, capabilities, platform |
Release version and compiled features; Windows reports managed PHP and Unix operations sockets unavailable |
instance_id |
Random process-local admin-instance identifier; changes on restart |
sampled_unix_secs, uptime_secs |
Sample wall time and monotonic uptime since admin initialization |
configuration.generation |
Starts at 1 for each instance; increments when this admin applies a runtime configuration, including rollback; not a persistent snapshot ID |
configuration.updated_unix_secs |
Time this configuration projection was published, not a collector heartbeat |
configuration.last_change |
Most recently completed reload/live rollback/self-heal rollback attempt that reached the operation, its outcome and time; null after restart |
health |
responding: the admin observer can respond; not an end-to-end application probe |
readiness |
ready (no pending recovery), validating, or recovery_required (expired validation, excessive failures or a failed rollback) |
consistency |
best_effort: individually safe samples, not a transaction across all workers |
stale |
True when the configuration inventory is incomplete; inspect inventory.complete before using counts to claim a feature is disabled |
Readiness describes admin recovery eligibility only. It does not prove that
every public listener, origin, PHP worker or certificate is healthy. A collector
failure does not make public serving unready. HTTP 200 means a sample is available;
clients must inspect readiness and the separate telemetry states.
Generation and last-change completion are separate observations. A failing apply
may already have changed the running configuration before persistence failed.
failed is not a promise of rollback or a durable audit record. A concurrent read
can see a new generation before the operation's completion record. Auth/audit
admission rejection before an operation does not replace its last-change record.
External certificate reloads, discovery and restart history are not represented
as admin configuration changes.
Inventory contains aggregate counts only: no vhost names, backend addresses,
paths, URLs, tokens, snapshot IDs, certificate identities or error messages.
listeners counts configured public HTTP/TLS listener entries, not successfully
bound sockets; it excludes admin, metrics, stream and UDP listeners. Upstream
items count explicit primary, list and backup entries, without deduplication or
DNS/discovery expansion. Cache, PHP and ACME count explicitly enabled sections,
not effective inherited routes, worker counts, stored objects or certificate count.
The summary visits at most 1,024 global/vhost/route scopes when a configuration
is published, never per status request. If truncated, counts describe only the
visited prefix and are lower bounds. In that case disabled means not configured
in that prefix, not globally disabled. vhosts remains the exact vector count.
The component state is not_compiled, disabled or configured.
Its runtime: unavailable is deliberate: this projection does not probe or
walk mutable subsystem inventories and must not invent healthy zero values.
Detailed existing routes remain available separately where needed.
Telemetry uses fixed fields and process-local counters:
- Logging and structured-event workers report
observedwhen alive,degradedwhen a worker has stopped, orunavailablebefore initialization. Failure counters are cumulative; pending records are not durable persistence. - Metrics/traces report
not_compiled,disabled,unavailablebefore a completed attempt,observedafter the most recent successful attempt, ordegradedafter partial/failed delivery. No new collector probe is sent. - Export counters are cumulative since process start, not per configuration. Queue bytes and active requests are instantaneous best-effort gauges. The most recent outcome can be old when there is no traffic. An observed outcome is not a current collector reachability guarantee.
- Unknown values are
null, not misleading zeros. Telemetry loss and degradation do not override the independent readiness field.
Collection uses only fixed-size cached summaries, atomic counters and nonblocking
state reads. At most four collectors run per admin instance. A busy/poisoned
state lock returns 503 operational_status_unavailable, without waiting.
Serialization is capped before buffer growth at 16 KiB. Collection checks a
50 ms elapsed-time budget before returning; OS scheduling and network delivery
are not hard real-time guarantees. The normal admin listener connection,
request-header/body and write limits still apply.
The new observer never loads snapshot files, probes origins, contacts collectors,
reopens logs, acknowledges validation, changes configuration or triggers rollback.
An expired/failed recovery remains visible as recovery_required. The watchdog
and all existing routes retain their previous recovery enforcement. Existing
mutations still cannot confirm a candidate past a failed rollback.
Errors: 400 unsupported query; 401 missing/incorrect token; 403 credential scope or certificate
policy; 405 method; 414 existing request-target bounds; 429 authentication
throttling; 503 unavailable/busy/over-budget sample. Retry 503 with client-side
backoff rather than a tight loop. No new CORS policy or unauthenticated exception
is introduced.
These routes remain compatible, but are not redefined by the new fixed-size OpenAPI contract. Authenticated HTTP reads can still enforce recovery before returning. Some read snapshot files or enumerate detailed runtime state; do not assume the new observer's 16 KiB/50 ms bounds apply to them.
| Method and route | Purpose |
|---|---|
GET /_fluxheim/status |
Existing snapshot/recovery, logging and feature-specific details |
GET /_fluxheim/cache/status |
Existing cache activity and configured tiers |
GET /_fluxheim/load-balancer/status |
Existing pool/member details |
GET /_fluxheim/udp/status |
Existing UDP status |
GET /_fluxheim/snapshots |
Existing persisted snapshot list |
GET /_fluxheim/health |
Configurable health path/mode; optionally unauthenticated; not the operational contract |
For existing mutations and their authorization/recovery behavior see snapshots and admin API, cache and load balancing. For the complete operation inventory, transport and role boundaries, see the admin guide. The observer credential is not a general read-only admin role.
Install the pinned development-only schema tooling in a virtual environment:
python3 -m venv target/status-venv
target/status-venv/bin/python -m pip install -r scripts/operational_status_requirements.txt
target/status-venv/bin/python scripts/test_operational_status_contract.py
cargo build --locked --no-default-features --features profile-observability,php-fpm --bin fluxheim
target/status-venv/bin/python scripts/smoke_operational_status.py --binary target/debug/fluxheimWindows uses target/status-venv/Scripts/python.exe and fluxheim.exe. The smoke
owns a private temporary fixture and local collector; it checks live auth/schema,
secret redaction, concurrency, telemetry outage/recovery and reload/rollback/restart.
--reduced tests a proxy,web build without OTLP. See the
checkpoint evidence for actual native results.