fix(providers): guard compaction against token estimate drift

AGENTS.md

@@ -228,7 +228,7 @@ step-by-step checklist.
 - **Hardened dashboard request surface** (issue #397): three independent layers, none individually load-bearing. (1) `web/auth.py::OriginHostGuard`, a pure-ASGI middleware (`app.add_middleware(...)`, not `BaseHTTPMiddleware`/`@app.middleware("http")` — neither sees WebSocket scopes) validating `Host` (required, must name the bound machine) and `Origin` (only checked when present — httpx/curl/`conductor gate respond` send none, and that path must keep working with no extra setup) on every HTTP and WebSocket request. `CONDUCTOR_WEB_ALLOW_ORIGINS` (comma-separated full origins) is the dev-server escape hatch, additive only. (2) A per-run token, minted automatically (`mint_token()`) so the protected configuration is the default; `CONDUCTOR_GATE_TOKEN` overrides it (`resolve_expected_token`). Required on every mutating route (`/api/stop`, `/api/kill`, `/api/resume`, `/api/gate-respond`, `/api/guidance`) and on the `/ws` handshake — the single auth point for the socket, since an unauthenticated connection is closed (`websocket.close`, code 1008) in reply to `websocket.connect`, before `accept()`, so it can never send *any* message type. This is strictly stronger than the pre-#397 per-message check, which browsers could never satisfy at all. Read-only routes (`/api/state`, `/api/info`, `/api/logs`, `/api/gate-status`, `/api/files/*`, the whole replay app) stay unauthenticated, protected by origin/host only. (3) `Content-Type: application/json` required on every mutating route (415 otherwise), including the bodyless control POSTs. **Token discovery:** `WebDashboard.start()` writes a `0600` file (POSIX; on Windows the mode is not honoured and the file relies on the user-profile NTFS ACL instead) at `~/.conductor/runs/dashboard-<port>.token` once the port resolves (works for both `--web` and `--web-bg`, since `cli/run.py` calls `start()`/`stop()` on both the run and resume paths); `stop()` removes it. `conductor gate respond`, `conductor guide`, and `conductor stop`'s graceful-kill rung all resolve a token via the shared `resolve_cli_token(port, token)`: `--token` > `CONDUCTOR_GATE_TOKEN` > the token file. See `docs/cli-reference.md` (Environment Variables, and the Authentication sections under `conductor gate respond` / `conductor guide`) and the `web/` bullets above for the full mechanism.
 - **MCP server exposure** (`mcp:` workflow block, `conductor mcp serve`, issue #432): `workflow.mcp` (`config/schema.py::McpConfig`) is a typed, `extra="forbid"` block — `expose` (default `true`; every workflow is a candidate for MCP tool exposure with no editing required, DD4), `mode` (`async`/`sync`/`auto`; the default a generated tool's omitted `_wait_seconds` resolves to), `read_only` / `destructive` (surfaced as the generated tool's `readOnlyHint`/`destructiveHint` annotations), and `estimated_minutes` (a client-side hint, must be positive). `conductor validate` reports an unknown key inside it as a schema error, not silence (FR11) — it cannot ride on the existing untyped `metadata: dict`. See `examples/mcp-serve.yaml` and `docs/mcp-server.md` (the user-facing guide: host configuration, the exposure ladder, toolsets, the run lifecycle, and a dedicated *Limits* section for DD5/DD9/DD11/DD12/R4) and `src/conductor/mcp/serve/` above for the server that reads it. **R1 — this feature's terminal run record is a scope change to `conductor status` and `conductor fleet list`, not an MCP-only addition**: every run — MCP-launched or not — now writes a completion tombstone, so both commands (and the Fleet TUI's History screen) gained a completed-runs section as a side effect, with `--live` restoring the exact pre-change scope. See the `fleet/records.py` (`TerminalRunRecord`) bullet above and the `CHANGELOG.md` entry for the full description of what changed.
 
-- **Context compaction**: Always-on client-side context window compaction for the `claude` and `openai` providers. Compaction is triggered proactively using the reserve-based formula `trigger = window - (output_limit + buffer)` and targets a clamped 55% hysteresis ceiling. In this formula, the `output_limit` resolves to the minimum of the effective `max_tokens` sent to the API (from settings or defaults) and the model output cap (from provider-cap). The tool-output-derived `buffer` is calculated as `2 * ceil(max_chars / 4) + 15,000` tokens as a heuristic for worst-case tool results. Resolution prioritizes cascades: provider-advertised metadata (with full pagination for Anthropic and a vendor-field parser for OpenAI-compatible endpoints), public registry, and conservative fallback. The wrapper operates in a fail-open manner, disabling itself for the rest of the agent execution upon outer failure. Compaction events are emitted to the dashboard and console. A summarizing compaction step runs a nested model call that consumes a request slot from the agent's `max_agent_iterations` budget, which doesn't get refunded. The dashboard context bar relies on provider-only limits and may disagree with the compaction window.
+- **Context compaction**: Always-on client-side context window compaction for the `claude` and `openai` providers. Compaction is triggered proactively using the reserve-based formula `trigger = window - (output_limit + buffer)` and targets a clamped 55% hysteresis ceiling. In this formula, the `output_limit` resolves to the minimum of the effective `max_tokens` sent to the API (from settings or defaults) and the model output cap (from provider-cap). The tool-output-derived `buffer` is calculated as `2 * ceil(max_chars / 4) + 15,000` tokens as a heuristic for worst-case tool results. Resolution prioritizes cascades: provider-advertised metadata (with full pagination for Anthropic and a vendor-field parser for OpenAI-compatible endpoints), public registry, and conservative fallback. A second, density-calibrated estimate guards the hard window against token-dense content (CJK, base64, minified data) the ~4-chars-per-token heuristic undercounts: it matches the heuristic on ordinary prose, so it never compacts a history that is merely large, and when it fires the tier chain is driven directly against that measurement rather than delegated to the inner strategy's own (heuristic) gate. Token telemetry stays on the primary scale — the density value travels as `density_tokens` and `trigger_reason` on the start event, with `degraded_estimators` / `still_over_window` on the complete event and an `agent_compaction_skipped` event when both estimators fail. The wrapper operates in a fail-open manner, disabling itself for the rest of the agent execution upon outer failure. Compaction events are emitted to the dashboard and console. A summarizing compaction step runs a nested model call that consumes a request slot from the agent's `max_agent_iterations` budget, which doesn't get refunded. The dashboard context bar relies on provider-only limits and may disagree with the compaction window.
 
 ### Debugging `--web-bg` failures
 

CHANGELOG.md

@@ -27,6 +27,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   entirely. See
   [`examples/claude-agent-sdk-setting-sources.yaml`](examples/claude-agent-sdk-setting-sources.yaml).
 
+### Fixed
+
+- **Context compaction window guard against token-dense drift** (#507) — the
+  `claude` / `openai` providers' compaction trigger anchors on
+  provider-reported token usage and estimates everything after the anchor
+  with a ~4-characters-per-token heuristic, which undercounts token-dense
+  content (CJK and other non-Latin scripts, base64, hex, minified data) by
+  2-4x. A dense suffix could therefore grow the real request past a known
+  context window while the trigger estimate stayed below the threshold, and
+  the provider rejected the request with `context_length_exceeded`. A second,
+  density-calibrated estimate now guards the hard window: it matches the
+  primary heuristic on ordinary prose, counts text with a substantial
+  non-ASCII share at ~1 token per character, and whitespace-poor ASCII blobs
+  at ~2 characters per token, so it fires only on genuinely dense content —
+  never on a history that is merely large. When it fires, the tier chain is
+  driven directly against that measurement (the inner strategy's own gate
+  would re-measure with the same heuristic that under-counted the content
+  and no-op), until the estimate is back under the target. Telemetry stays on
+  the token scale: `agent_compaction_start` gains `trigger_reason`
+  (`"trigger"` / `"window_guard"`) and a separate `density_tokens` field
+  instead of overloading `tokens_before`, and `agent_compaction_complete`
+  gains `degraded_estimators` and `still_over_window` so a guard compaction
+  that could not get back under the window reads as degraded, not as false
+  success. A failed primary measurement falls back to an independent
+  density-calibrated estimate that shares no code with it, and a double
+  failure is reported as a new `agent_compaction_skipped` event
+  (`reason: "estimate_unavailable"`) rather than vanishing into stderr. See
+  [Workflow Syntax → Context Compaction](docs/workflow-syntax.md#context-compaction).
+
 ## [0.1.36](https://github.com/microsoft/conductor/compare/v0.1.35...v0.1.36) - 2026-09-02
 
 ### Added

docs/workflow-syntax.md

@@ -2425,6 +2425,14 @@ Below is how these values resolve in practice for different configurations using
 
 When the reserve (output limit plus effective tool buffer) leaves no viable headroom below the window, compaction is disabled for the agent execution rather than armed with a degenerate threshold. The `agent_compaction_config` event then carries `enabled: false` and a `disabled_reason`, so the condition is visible per run instead of surfacing as a one-shot log warning. To resolve this, lower `runtime.max_tokens` or `tool_output.max_chars`.
 
+#### Window Guard Against Token-Dense Content
+
+The trigger is measured by the primary estimator: the provider's reported token usage for the history up to the most recent response, plus a ~4-characters-per-token heuristic for everything after it. That heuristic undercounts token-dense content — CJK and other non-Latin scripts, base64, hex, or minified data — by 2-4x, so a dense suffix can grow the real request past the known context window while the trigger estimate stays below the threshold.
+
+A second, density-calibrated estimate guards the hard window. It matches the primary heuristic on ordinary prose, counts text with a substantial non-ASCII share at ~1 token per character, and whitespace-poor ASCII blobs at ~2 characters per token. When that estimate reaches the known window, compaction runs even if the trigger never fired, and the tier chain is driven against the density-calibrated measurement until the history fits the target. Because the two estimates agree on ordinary text, the guard never compacts a history that is merely large.
+
+The start event reports which gate fired via `trigger_reason` (`"trigger"` or `"window_guard"`) and carries the density-calibrated value separately as `density_tokens`; `tokens_before` always stays the primary token estimate.
+
 ### Compaction Tiers
 
 Conductor uses three sequential tiers to compress the history down to the target:
@@ -2462,12 +2470,13 @@ All tokens consumed by summarizing compaction are added to the workflow's total
 
 ### Observability and Events
 
-Compaction operates in a fail-open manner. If an error occurs during compaction, Conductor logs a warning, disables compaction for the rest of that agent's execution, and continues with the uncompacted history.
+Compaction operates in a fail-open manner. If an error occurs during compaction, Conductor logs a warning, disables compaction for the rest of that agent's execution, and continues with the uncompacted history. A failed context measurement never disables anything: the primary estimate falls back to an independent density-calibrated one, and only when both fail is compaction skipped for that request alone, reported as `agent_compaction_skipped` with `reason: "estimate_unavailable"`.
 
-Conductor emits three event types to track compaction:
+Conductor emits four event types to track compaction:
 *   `agent_compaction_config`: Emitted once at the start of agent execution to log resolved window and limit values.
-*   `agent_compaction_start`: Emitted when context size exceeds the trigger threshold and compaction begins.
-*   `agent_compaction_complete`: Emitted when compaction completes, detailing token savings or errors.
+*   `agent_compaction_start`: Emitted when compaction begins. `trigger_reason` names the gate that fired (`"trigger"` or `"window_guard"`), and `density_tokens` carries the density-calibrated estimate alongside the primary-scale `tokens_before`.
+*   `agent_compaction_complete`: Emitted when compaction completes, detailing token savings or errors. Degraded outcomes are named rather than hidden: `degraded_tiers` for recovered tier failures, `degraded_estimators` for lost measurements, `still_over_trigger` when the history remains above the trigger, and `still_over_window` when a window-guard compaction could not get back below the known window.
+*   `agent_compaction_skipped`: Emitted when compaction did not run because the context size could not be measured at all.
 
 ### Dashboard Caveat
 

src/conductor/cli/run.py

@@ -1242,13 +1242,19 @@ def on_event(self, event: WorkflowEvent) -> None:
                 messages_after = d.get("messages_after", 0)
                 elapsed = d.get("elapsed", 0.0)
                 degraded_tiers = d.get("degraded_tiers") or []
+                degraded_estimators = d.get("degraded_estimators") or []
                 still_over_trigger = d.get("still_over_trigger", False)
-                if degraded_tiers or still_over_trigger:
+                still_over_window = d.get("still_over_window", False)
+                if degraded_tiers or degraded_estimators or still_over_trigger or still_over_window:
                     reasons: list[str] = []
                     if degraded_tiers:
                         reasons.append(f"tier(s) degraded: {', '.join(degraded_tiers)}")
+                    if degraded_estimators:
+                        reasons.append(f"estimator(s) degraded: {', '.join(degraded_estimators)}")
                     if still_over_trigger:
                         reasons.append("history remains above the trigger")
+                    if still_over_window:
+                        reasons.append("history remains above the known context window")
                     verbose_log(
                         styled(
                             "  WARNING: context compacted for '[bold]{}[/bold]': "
@@ -1277,6 +1283,17 @@ def on_event(self, event: WorkflowEvent) -> None:
                         )
                     )
 
+        elif t == "agent_compaction_skipped":
+            verbose_log(
+                styled(
+                    "  WARNING: compaction skipped for '[bold]{}[/bold]' ({}) — "
+                    "context size could not be measured for this request",
+                    d.get("agent_name", "?"),
+                    d.get("reason", "unknown"),
+                ),
+                style="yellow",
+            )
+
         elif t == "guidance_received":
             pending = d.get("pending", 1)
             verbose_log(