Architecture Map

Quick re-orientation guide for the Compass multi-agent routing optimizer.

1. Pipeline Overview

graph TD
    U["User"] -->|problem + dataset| A1["Stage 1 · User Input Agent<br/><em>LLM-driven</em>"]
    A1 -->|validated_input_report_path| A2["Stage 2 · Data Validation Agent<br/><em>LLM-driven</em><br/>Phase 1: ingest &amp; map → Phase 2: validate + split"]
    A2 -->|RoutingContext + dev/holdout splits| A2b["Stage 3 · Backend Setup Agent<br/><em>LLM-driven</em><br/>write backend profile"]
    A2b -->|backend profile| A3["Stage 4 · Prompt Builder Agent<br/><em>LLM-driven</em>"]
    A3 -->|prompt version| A4["Stage 4 · Eval Runner Agent<br/><em>code-driven</em>"]
    A4 -->|eval_score_report| A5["Stage 4 · Review Agent<br/><em>LLM-driven</em>"]
    A5 -->|loop_signal = refine| A3
    A5 -->|loop_signal = exit / converged| A6["Stage 5–6 · Final Report Agent<br/><em>Hybrid (code + LLM)</em><br/>Holdout eval + report"]
    A6 -->|final report| U

Stage numbering. The six agent roles above map to five internal dispatcher stages in compass/agents/pipeline/status.py (_STAGES): Stage 4 covers the Prompt Builder ↔ Eval Runner ↔ Review Agent loop, and internal Stage 5 ("Final Report") runs both holdout evaluation and report generation. The README's ## Usage section presents the same flow as six user-facing stages.

Rerun mode: When Stage 4 has converged, the orchestrator can call initiate_rerun to re-enter the pipeline at Stage 3 for a different backend. The rerun flow is: Stage 3 (new backend) → Stage 4 (Prompt Builder Rerun: format restructure + single eval) → Stage 5 (final report). The original search state is preserved as search_state_original.json; rerun_config.json drives rerun-mode behavior throughout status.py.

2. Agent Registry

Agent	Type	Module / Prompt	Status	Reads from Context	Writes to Context
User Input	LLM-driven	`compass/agents/prompts/user_input_system.md`, `compass/agents/user_input/report.py`	Done	(user conversation)	`validated_input_report_path`
Data Validation	LLM-driven	`compass/agents/prompts/data_validation_system.md`, `compass/agents/data_validation/checks.py`, `compass/agents/data_validation/split.py`	Done	`validated_input_report_path`	`data_quality_report`, `routing_context`, `dataset_path`, `original_dataset_path`, `dev_jsonl_path`, `holdout_jsonl_path`, `split_report_path` (debug-only)
Eval Runner	Code-driven	`compass/agents/eval_runner.py`	Done	`prompt_version`, `data_source`, `backend`, `run_config` or `config_path`	`eval_score_report`
Backend Setup	LLM-driven	`compass/agents/prompts/backend_setup_system.md`	Done	(user conversation)	`backend` (new YAML file written to `backends/`)
Prompt Builder	LLM-driven	`compass/agents/prompts/prompt_builder_system.md`, `compass/agents/prompt_builder/search.py`, `search_ops.py`	Done	`routing_context`, `dev_jsonl_path`, child variants	`prompt_version` (one file per candidate)
Prompt Builder Rerun	LLM-driven	`compass/agents/prompts/prompt_builder_rerun_system.md`	Done	`run_id`, `source_prompt_version`, `new_backend` (from subagent instruction)	`prompt_version` (restructured)
Review	Hybrid (code + LLM)	`compass/agents/review/models.py`, `compass/agents/review/preprocessor.py`, `compass/agents/review/ops.py`, layered prompt: `review_agent_base_system.md` + phase base + beam overlay (see Prompts table)	Done	`eval_score_report`, `review_briefing`	`review_result` (debug-only)
Final Report	Hybrid (code + LLM)	`compass/agents/final_report/models.py`, `compass/agents/final_report/preprocessor.py`, `compass/agents/prompts/final_report_system.md`, `compass/agents/prompts/final_report_template.md`	Done	holdout dataset, search state, all eval reports	`final_report.md`, per-version `baseline_comparison.json`, optimization charts

3. Context Dict Reference

Run scoping: All pipeline artifacts are scoped to a run_id generated by submit_input_report. Context keys below correspond to files persisted under outputs/<run_id>/. The run_id is passed explicitly to all MCP tools.

Key	Type	Set By	Consumed By	Description
`validated_input_report_path`	`str`	User Input Agent	Data Validation Agent	Filesystem path to the Markdown input report
`eval_score_report`	`ScoreReport`	Eval Runner Agent	Review Agent	Metrics, summary, error breakdown, and run-over-run diff
`prompt_version`	`str`	Prompt Builder Agent / MCP tool param	Eval Runner Agent	Prompt version identifier (e.g. `"v3"`, `"latest"`)
`data_source`	`str`	MCP tool param	Eval Runner Agent, Data Validation Agent	Path to the JSONL dataset file
`backend`	`str`	MCP tool param	Eval Runner Agent	Backend label matching a profile in `backends/`
`config_path`	`str`	MCP tool param	Eval Runner Agent	Path to YAML run config (default `outputs/run_config.yaml`)
`routing_context`	`RoutingContext`	Data Validation Agent	Prompt Builder Agent	Domain-agnostic routing config: routes, dimensions, ordering, seed vocabulary
`data_quality_report`	`DataQualityReport`	Data Validation Agent	(diagnostics)	Schema findings, label distribution, volume assessment
`dataset_path`	`str`	Data Validation Agent	(provenance tracking)	Path to validated JSONL dataset before splitting
`original_dataset_path`	`str`	Data Validation Agent	(provenance tracking)	Path to the user's original dataset file before transformation
`dev_jsonl_path`	`str`	Data Validation Agent	Prompt Builder Agent	Dev split examples path
`holdout_jsonl_path`	`str`	Data Validation Agent	Final Reporting Agent	Holdout split examples path
`split_report_path`	`str \| null` (debug-only)	Data Validation Agent	Prompt Builder Agent	Split statistics and distribution report when debug artifact writing is enabled
`review_briefing`	`str` (markdown summary)	Review Agent (pre-processor)	Review Agent (LLM)	Progressive-disclosure briefing summary rendered from `ReviewBriefing`: round summary, routing context, per-class recall, target/confusion analysis, candidates, elite set, directive history
`review_result`	`ReviewResult` (debug-only)	Review Agent (LLM)	Prompt Builder Agent	Ranked candidates, edit directives, promotion decisions, loop signal, regression guards

4. Shared Models

Candidate / SearchState / RoundSummary (compass/agents/prompt_builder/search.py) Candidate is the canonical prompt-candidate record. Core fields: prompt_version, parent_version, quality_score, cost, round_introduced, example_ids. Optional fields (all default None): secondary_parent_version (two-parent merges), eval_status (parallel eval tracking), mutation_strategy, route_metrics. trajectory_id is present for other search strategies and unused by beam. Accepts iteration_introduced as an alias for round_introduced (back-compat). Old state files carrying dominated load without error (extra="ignore").

SearchState holds the mutable search loop state. Key fields:

Field	Type	Description
`elite_set`	`list[Candidate]`	Current non-dominated candidate set (formerly `pareto_front`; old files with `pareto_front` key are migrated on load)
`algorithm`	`str`	Discriminator set from `_BRANCH_ALGORITHM` at `init_search_state` time; this repo sets `"beam"`
`algorithm_state`	`dict[str, Any]`	Strategy-specific sub-state pocket, set from `_BRANCH_ALGORITHM_STATE` in `search_ops.py` (`{"beam_width": 3}`)
`round`, `stagnation_count`, `mutation_mode`	—	Search-loop bookkeeping
`converged`, `loop_phase`	—	Convergence flag and current sub-phase. Beam uses `"build"`, `"review"`, `"build_recovering"`; the enum also carries reserved values (`"calibration"`, `"warmup_seed"`, `"warmup_build"`, `"warmup_reduce"`) for other search strategies
`active_evals`	`list[str]`	Prompt versions currently being evaluated (pending or running). Invariant: non-empty iff `loop_phase == "build"` and a concurrent batch eval was interrupted. `_detect_stage_4_phase` returns `"build_recovering"` when this field is non-empty.

RoundSummary is the per-round progress record. A few fields were renamed from earlier names, which still load via a model_validator(mode="before"):

Field	Legacy alias
`new_elite_entries`	`new_pareto_points`
`elite_size`	`front_size`
`target_improvement`	`front_improvement`

mutation_mode and stagnation_count are optional (None when unset).

DataQualityReport (compass/agents/data_validation/checks.py) Top-level report from the Data Validation agent containing SchemaFinding list, LabelDistribution, VolumeAssessment, and optional QueryLengthDistribution. The LLM agent writes the narrative summary; the Python checks populate the structured sections.

RoutingContext (compass/agents/routing_context.py) Domain-agnostic routing configuration holding a domain description, RouteDefinition list, RoutingDimension list, optional RouteOrdering, and optional SeedVocabulary. Produced by the Data Validation Agent and consumed by the Prompt Builder Agent.

ReviewBriefing / ReviewResult (compass/agents/review/models.py) ReviewBriefing is the complete pre-processed input for the Review Agent LLM, containing CandidateAnalysis list, DiversityMetrics, DiminishingReturns, OracleMetrics, per-class recall, UserTargetProgress list (progress toward user-specified metric targets; each entry carries source_version — the single candidate version evaluated, all entries sharing the same value), single_candidate_meets_all flag (true when every target is met by the same candidate — the only safe condition for LoopSignal{action="exit"}), BatchOutcome list (linking child variants to their eval results), directive_history list (DirectiveOutcome entries for prior-round directives — synthesized wholly in code by _synthesize_directive_outcomes inside build_review_briefing from batch_outcomes; no directive_history.json file is persisted and the agent no longer writes outcomes), ChildVariant list, and initial_parent_version (canonical parent_version for cold-start / warm-up seeds; default "base"). Fields are ordered stable-first, varying-last for prompt-cache prefix stability (see render.py).

build_review_briefing now returns a markdown progressive-disclosure summary (rendered by compass/agents/review/render.py) rather than raw JSON. Companion detail-fetch tools (get_score_report, get_confusion_cell, get_round_child_variants, query_dev_examples, query_holdout_examples, get_dataset_oracle_distribution, get_per_class_recall) provide on-demand drill-down without reloading the full briefing on every call. Dataset rows remain query-only: row tools page results with offset/limit and never inline an entire dataset. get_directive_history and get_batch_outcomes are retained only as deprecated MCP placeholders because the legacy round_reports persistence they depended on is no longer written by production code. ReviewResult is the LLM output: candidate_ranking, child_variants, promotion_decisions, loop_signal, and regression_guards. Review-side persistence and legacy artifact loading live in review/ops.py. Child variants are persisted to child_variants.json via record_directive_outcomes and retrieved by the Prompt Builder via get_child_variants. get_edit_directives is a back-compat helper that flattens all directives across variants into a single list.

ReviewBriefing also carries optional None-default slots, populated by the round-advance preprocessor:

Field	Type	Used by
`parent_a_version`, `parent_b_version`	`str \| None`	beam — set by `build_review_briefing` for two-parent merges
`stagnation_signal`	`dict \| None`	beam — hypervolume-delta cue for the iterative overlay
`trajectory_id`, `weight_vector`, `binding_axis`, `acceptance_history`	—	other search strategies only; `None` under beam

The round-advance logic for beam search (advance_round_beam) lives in search_ops.py.

ScoreReport / RunReport (compass/eval/models.py) RunReport is the full evaluation output (config, metrics, results, summary). ScoreReport is the inter-agent contract (context key eval_score_report) containing metrics, summary, error breakdown, run-over-run RunDiff, and output file paths.

BackendProfile (compass/eval/backends/profile.py) Pydantic model representing a validated backend configuration loaded from a YAML file in backends/. Key fields:

Field	Type	Description
`provider`	`Literal["anthropic", "openai", "bedrock", "mock_echo"]`	SDK backend selector
`model`	`str`	Model identifier (e.g. `"claude-sonnet-4-20250514"`)
`requests_per_minute`	`int`	RPM rate limit cap
`tokens_per_minute`	`int`	TPM rate limit cap
`max_tokens`	`int \| None`	Max tokens to generate
`temperature`	`float \| None`	Sampling temperature
`reasoning_level`	`Literal["low", "medium", "high"] \| None`	Reasoning effort/budget tier; mapped per-provider (Anthropic: `thinking.budget_tokens`, OpenAI: `reasoning_effort`)
`pricing`	`ModelPricing \| None`	Inline cost config for token-based cost tracking
`api_key_env`	`str \| None`	Env var name holding the API key
`extra_params`	`dict[str, Any]`	Additional kwargs splatted into the provider SDK's `create()` call
`provider_params`	`dict[str, Any]`	Provider-specific kwargs for client construction

5. MCP Surface

Tools

Name	Status	Purpose	Backing Module
`optimize_routing_prompt`	Stub	Run the full routing prompt optimization pipeline	`compass/mcp/`
`run_batch_eval`	Implemented	Evaluate one or more prompt candidates concurrently; `candidates=[]` triggers recovery mode.	`compass/eval/batch_eval.py`
`run_holdout_eval`	Implemented	Run evaluation on the holdout split with one or more Pareto-front prompt versions (Final Report agent only). Also computes baseline comparisons and returns an artifact manifest (`report_path`, `results_path`, optional `baseline_comparison_path`) for the next `build_final_report_briefing` step.	`compass/mcp/final_report_tools.py`
`submit_input_report`	Stub	Submit a validated input report to the pipeline	`compass/mcp/`
`validate_dataset`	Implemented	Run all validation checks against a JSONL routing dataset	`compass/agents/data_validation/checks.py`
`detect_and_parse_dataset`	Implemented	Detect format and parse a raw dataset file; accepts `run_id`	`compass/agents/data_validation/detect.py`
`transform_dataset`	Implemented	Apply column mappings to normalize a dataset to the canonical schema; rejects mappings whose output violates `expected.route ∈ expected.routes.keys()`	`compass/agents/data_validation/transform.py`
`stratified_split`	Implemented	Split dataset into dev/holdout	`compass/agents/data_validation/split.py`
`get_routing_context`	Implemented	Return a markdown summary of the RoutingContext for a run; reads `outputs/<run_id>/validation/routing_context.json`; also available to the Prompt Builder sub-agent	`compass/mcp/data_validation_tools.py`
`build_review_briefing`	Implemented	Pre-process a round's candidates into a ReviewBriefing and return its markdown progressive-disclosure summary for the Review Agent	`compass/mcp/review_tools.py`
`record_directive_outcomes`	Implemented	Persist child variants and the Review Agent `loop_signal`; directive outcomes are synthesized later in code during `build_review_briefing` rather than written as a standalone artifact	`compass/mcp/review_tools.py`
`query_dev_examples`	Implemented	Query dev examples, optionally filtered by route, with `offset`/`limit` pagination for stage-4 review work	`compass/mcp/review_tools.py`
`query_holdout_examples`	Implemented	Query holdout examples, optionally filtered by route, with `offset`/`limit` pagination for directive crafting	`compass/mcp/review_tools.py`
`get_prompt_text`	Implemented	Retrieve the full text of a versioned prompt; requires `run_id`; checks `outputs/<run_id>/prompts/` first, falls back to project-level `prompts/`; also available to the Prompt Builder sub-agent	`compass/mcp/review_tools.py`
`get_score_report`	Implemented	Return markdown ScoreReport for a candidate version via the shared renderer: metrics table, summary, top-K errors, diff status; also available to the Prompt Builder sub-agent	`compass/mcp/review_tools.py`
`get_confusion_cell`	Implemented	Return markdown detail for a single confusion cell (recomputes briefing, slices matching ConfusionImpact)	`compass/mcp/review_tools.py`
`get_directive_history`	Implemented (deprecated)	Returns a deprecation message; the historical data source it depended on was retired with `round_reports` writer removal. Use `build_review_briefing` for recent outcomes and `get_score_report` for per-version detail.	`compass/mcp/review_tools.py`
`get_batch_outcomes`	Implemented (deprecated)	Returns a deprecation message; the historical data source it depended on was retired with `round_reports` writer removal. Use `build_review_briefing` for recent outcomes and `get_score_report` for per-version detail.	`compass/mcp/review_tools.py`
`get_round_child_variants`	Implemented	Return markdown of child variants grouped by variant_id; optionally includes full directive bodies	`compass/mcp/review_tools.py`
`get_dataset_oracle_distribution`	Implemented	Return per-route oracle aggregates from `analysis/dev.jsonl` (n_labeled, mean cost/quality, Pareto-optimal count, ties-with-cheaper-route count); optional row-level drill-down filtered by route or example_ids	`compass/mcp/review_tools.py`
`get_per_class_recall`	Implemented	Return the full per-class recall table for a round (all routes, no median/regression filter)	`compass/mcp/review_tools.py`
`build_final_report_briefing`	Implemented	Pre-process all pipeline artifacts into a structured briefing with charts for the Final Report Agent	`compass/agents/final_report/preprocessor.py`
`save_final_report`	Implemented	Save the final report markdown to disk	`compass/mcp/final_report_tools.py`
`get_pipeline_status`	Implemented	Read-only inspector for pipeline progress. Not part of the dispatch loop — orchestrators use `start_stage` directly.	`compass/agents/pipeline/status.py`
`get_default_pricing`	Implemented	Look up default pricing for a (provider, model) pair; used by the backend setup agent	`compass/eval/pricing.py`
`init_search_state`	Implemented	Initialize prompt-builder search state for a run; the algorithm is set from the module constants in `search_ops.py` — no `algorithm`/`algorithm_state` params	`compass/agents/prompt_builder/search_ops.py`
`register_candidate`	Implemented	Register a new prompt candidate for evaluation	`compass/agents/prompt_builder/search_ops.py`
`record_eval_result`	Implemented	Record evaluation results for Pareto tracking	`compass/agents/prompt_builder/search_ops.py`
`advance_step`	Implemented	Advance the search by one round; dispatches to `advance_round` → `advance_round_beam` in `search_ops.py`	`compass/mcp/prompt_building_tools.py`
`get_search_state`	Implemented	Return a markdown summary of the current search state via the shared renderer, with round history capped to the last 3 rounds	`compass/mcp/prompt_building_tools.py`
`filter_holdout_dataset`	Implemented	Remove few-shot examples from holdout before final eval	`compass/agents/prompt_builder/holdout_filter.py`
`get_child_variants`	Implemented	Retrieve the current round's child variants (grouped directives per child prompt) for the Prompt Builder	`compass/mcp/prompt_building_tools.py`
`get_edit_directives`	Implemented	Back-compat helper: flattens all directives across child variants into a single list	`compass/mcp/prompt_building_tools.py`
`save_prompt`	Implemented	Save compiled routing prompt to disk with correct encoding	`compass/mcp/prompt_building_tools.py`
`start_stage`	Implemented	Single dispatch verb. Inspects artifacts to choose the next stage, activates it, and returns the sub-agent prompt + dispatch checklist + recommended model in one payload.	`compass/mcp/orchestrator_tools.py`
`complete_stage`	Implemented	Reset to orchestrator scope after a sub-agent finishes	`compass/mcp/orchestrator_tools.py`
`initiate_rerun`	Implemented	Validate Stage 4 is complete, select best prompt version, rename search state, write `rerun_config.json`	`compass/mcp/orchestrator_tools.py`

Stage-Scoped Tool Filtering

The orchestrator calls start_stage(run_id) before spawning a sub-agent (no stage argument — the server infers the next stage from artifacts) and complete_stage(run_id) when it returns. While a stage is active, tools/list returns only the tools in STAGE_REGISTRY[stage] (defined in compass/mcp/server.py). This prevents sub-agents from calling tools outside their scope.

Stage	Tools
`orchestrator`	`optimize_routing_prompt`, `get_pipeline_status`, `start_stage`, `complete_stage`, `initiate_rerun`
`input_report`	`submit_input_report`, `get_pipeline_status`
`data_validation`	`detect_and_parse_dataset`, `transform_dataset`, `validate_dataset`, `save_routing_context`, `get_routing_context`, `stratified_split`, `get_pipeline_status`
`backend_setup`	`get_default_pricing`, `get_pipeline_status`
`prompt_building`	`init_search_state`, `register_candidate`, `run_batch_eval`, `record_eval_result`, `advance_step`, `get_search_state`, `get_routing_context`, `get_edit_directives`, `get_child_variants`, `get_prompt_text`, `get_score_report`, `save_prompt`, `signal_eval_complete`, `get_pipeline_status`
`review_cold`	`build_review_briefing`, `record_directive_outcomes`, `query_dev_examples`, `get_search_state`, `get_score_report`, `get_confusion_cell`, `get_round_child_variants`, `get_dataset_oracle_distribution`, `get_per_class_recall`, `get_pipeline_status`
`review`	`build_review_briefing`, `record_directive_outcomes`, `query_dev_examples`, `query_holdout_examples`, `get_prompt_text`, `get_search_state`, `get_score_report`, `get_confusion_cell`, `get_round_child_variants`, `get_dataset_oracle_distribution`, `get_per_class_recall`, `get_pipeline_status`
`calibration`	Reserved phase for other search strategies; not used by beam
`final_report`	`filter_holdout_dataset`, `run_holdout_eval`, `build_final_report_briefing`, `save_final_report`, `get_pipeline_status`

Model Routing

optimize_routing_prompt and get_pipeline_status embed two-layer model-routing hints. The source of truth is _REVIEW_AGENT_PROMPT_NAMES; the resolver is recommended_model_for.

Layer 1 — Universal capability claim (all consumers):

Stage / dispatch prompt	Tier	Rationale
`compass_review_agent_iterative`, `compass_review_agent_cold_start`	strong	High-stakes synthesis: confusion deltas, hypothesis generation
All other stages (`input_report`, `data_validation`, `prompt_building`, `final_report`)	fast	Tool-driven / rote tasks; 3× cheaper with no observed quality regression

Layer 2 — Claude Code binding (Claude Code consumers only — ignore otherwise):

Every Agent({...}) call MUST include a literal model parameter. Omitting it inherits the orchestrator's model (Sonnet under auto mode), silently violating the routing rule. Required aliases: model: "sonnet" for review/review_cold, model: "haiku" for all other stages. Each start_stage response carries a recommended_model field with the correct value for the current dispatch. If a Claude Code installation lacks one of these aliases, fall back to the closest available tier and report it in the run summary.

Non-Claude-Code MCP consumers see both layers as plain text and should map the tier to whatever their backend offers. No model defaults inside compass/* are changed.

Sub-Agent Guard Pattern

start_stage implements a two-sided entry/exit contract for sub-agent dispatch:

Gate point	Side	Mechanism	Strength
Entry	Orchestrator	`<HARD_STOP>` in `start_stage`'s `subagent_instruction`; sub-agent prompt in `sub_agent_prompt`	Hard (structural)
Entry	Sub-agent	`get_pipeline_status` call; stage mismatch → abort	Hard (behavioural)
Exit	Sub-agent	`get_pipeline_status` call; incomplete → fix before exiting	Hard (behavioural)
Exit	Orchestrator	`<HARD_STOP>` post-exit instruction	Soft (advisory)

Each stage system prompt (stages 1–5) includes mandatory ## Entry verification and ## Exit verification blocks. Sequencing knowledge lives exclusively in pipeline/status.py; stage prompts know only their own stage number.

Pipeline guards (dispatch markers and fanout tracking)

The shared guard layer lives in compass/agents/pipeline/dispatch.py and is wired into orchestrator_tools.py and the Stage-4 tool bodies.

loop_phase enum — widened superset

SearchState.loop_phase accepts the following values:

Value	Used by
`"review"`	all strategies (default)
`"build"`	all strategies
`"warmup_seed"`, `"warmup_build"`, `"warmup_reduce"`	reserved for other search strategies — warmup phases
`"calibration"`	reserved for other search strategies — cold-start calibration phase
`"build_recovering"`	entered when `active_evals` is non-empty (interrupted batch eval). Recovery sub-agent calls `run_batch_eval(candidates=[])` to resume.
`"review_post_coldstart"`	derived phase (not stored on disk); `_detect_stage_4_phase` returns it for the round-2 post-cold-start review.

Beam uses "build", "review", "build_recovering", and the derived "review_post_coldstart". The remaining values are reserved for other search strategies; unknown values in legacy JSON are silently remapped to "review" by a model_validator(mode="before").

Dispatch markers

Two JSON sentinel files signal that a sub-agent is in-flight for the current round:

File	Written by	Cleared by	Guards
`search/build_dispatched.json`	`register_candidate` (first builder action)	`advance_step` (round complete); `run_batch_eval_impl` (when `active_evals` drains after batch eval)	`complete_stage("prompt_building")` rejects while present
`search/review_dispatched.json`	`build_review_briefing` (reviewer dispatch)	`record_directive_outcomes` (directives saved)	`complete_stage("review")` checks fanout

build_dispatched.json contains {"round": N}. review_dispatched.json is {"round": N}. (A search strategy that dispatches several parallel reviewers per round would extend this; beam uses one reviewer.)

Multi-trajectory review fanout (extension point)

_next_action_for_stage_4 can be extended to dispatch N Review Agent sub-agents in parallel — one per trajectory — for a search strategy that needs it. Beam uses a single-slot (expected=1) fanout; complete_stage("review") calls review_fanout_status(run_id, expected=1) and checks is_complete.

DispatchFanout and review_fanout_status

complete_stage("review") calls review_fanout_status(run_id, expected=1) (single-slot fanout). The function returns a DispatchFanout dataclass:

Attribute	Type	Meaning
`expected`	`int`	Number of Review Agent sub-agents expected this round
`completed`	`list[int]`	Slot ids that finished (wrote `child_variants.json`)
`in_flight`	`list[int]`	Dispatched but not yet finished
`not_dispatched`	`list[int]`	Not yet dispatched
`is_complete`	`bool`	`len(completed) >= expected`
`missing`	`list[int]`	`in_flight + not_dispatched`

For expected=1 (single-slot — what beam uses), fanout is complete when search/child_variants.json exists. Multi-slot dispatch is an extension point: review_fanout_status would be extended for a strategy that needs it.

child_variants.json

Written by record_directive_outcomes; acts as the canonical review-completion sentinel. (Per-trajectory variants child_variants_t<N>.json are an extension point for a multi-trajectory strategy.)

Defense-in-depth phase flip

_detect_stage_4_phase in status.py adds a safety net: if loop_phase == "build" but neither child_variants.json nor build_dispatched.json exist on disk, the phase is re-interpreted as "review". This prevents deadlock if markers are lost (e.g. mid-run server restart).

signal_eval_complete

Retained as a back-compat shim for runs paused before automated marker clearing was deployed. Clears build_dispatched.json and flips loop_phase to "review" if they have not already been cleared/flipped. Calling it in normal operation is a no-op.

Prompts

Name	Purpose	Backing File
`compass_routing_input`	Activate the User Input agent conversation	`compass/agents/prompts/user_input_system.md`
`compass_data_validation`	Activate the Data Validation agent conversation	`compass/agents/prompts/data_validation_system.md`
`compass_prompt_builder`	Activate the Prompt Builder agent (Stage 4 build phase)	`compass/agents/prompts/prompt_builder_system.md`
`compass_review_agent_iterative(algorithm)`	Review Agent — iterative phase (round ≥ 3); assembled from three layers: base + iterative phase base + beam overlay	see Review Agent prompt files below
`compass_review_agent_cold_start(algorithm)`	Review Agent — cold-start / seeding phase (round 1); assembled from three layers: base + cold-start phase base + beam overlay	see Review Agent prompt files below
`compass_review_agent_post_coldstart(algorithm)`	Review Agent — round-2 post-cold-start phase; assembled from three layers: base + post-coldstart override + iterative phase base	see Review Agent prompt files below
`compass_backend_setup`	Backend setup agent — select or create backend	`compass/agents/prompts/backend_setup_system.md`
`compass_final_report`	Final Report agent — holdout eval + report generation	`compass/agents/prompts/final_report_system.md`
`compass_prompt_builder_rerun`	Prompt Builder Rerun agent — format-only restructure for a different backend (single eval round)	`compass/agents/prompts/prompt_builder_rerun_system.md`

Review Agent prompt files — layered structure

The Review Agent prompt is assembled at dispatch time (see assemble_review_prompt) from a shared base, a phase-specific base, and a beam overlay — three layers for the iterative and cold-start phases. Round 2 uses base + post-cold-start override + iterative base. The algorithm argument on the MCP prompt selects the overlay (beam in this repo).

Dispatch protocol. The Stage-4 dispatcher (any session driving the MCP) MUST fetch the canonical system prompt via compass_review_agent_cold_start(algorithm) for the seeding round (round 1), compass_review_agent_post_coldstart(algorithm) for round 2, and compass_review_agent_iterative(algorithm) for rounds ≥ 3, and pass the returned text verbatim as the sub-agent system prompt. Hand-rolling the dispatch prompt is forbidden: the ChildVariant / EditDirective schema is declared with extra="forbid" (compass/agents/review/models.py:272-310) and cannot be safely re-stated, and the canonical prompt owns the no-Bash invariant (review_agent_base_system.md) needed to keep the sub-agent on the MCP surface.

File	Role
`review_agent_base_system.md`	Shared base — entry verification, briefing schema, directive types, output schema, self-check rules
`review_agent_iterative_base_system.md`	Iterative phase base — "identify failure mode → hypothesise → create directive" flow
`review_agent_cold_start_base_system.md`	Cold-start phase base — "formulate diverse strategies" flow
`review_agent_cold_start_overlay_beam.md`	Cold-start overlay — K = `beam_width` seeds, diversity across confusion cells and cost regions
`review_agent_iterative_overlay_beam.md`	Iterative overlay (round ≥ 3) — child allocation modes, confusion-cell ranking, stagnation cue
`review_agent_post_coldstart_overlay_beam.md`	Round-2 override — one protected child per cold-start seed, no merges

Resources

URI	Purpose	Backing File
`compass://backends/{backend_label}`	Backend profile YAML (resource template) — provider detection for prompt builder	`backends/{backend_label}.yaml` (user project dir)
`compass://agents/prompt-builder/best-practices`	General prompt engineering principles for routing prompts	`compass/agents/prompt_builder_best_practices.md`
`compass://agents/prompt-builder/conventions-claude`	Claude conventions and Anthropic cookbook patterns for routing prompts	`compass/agents/prompt_builder_conventions_claude.md`
`compass://agents/prompt-builder/conventions-openai`	OpenAI GPT-5 conventions and cookbook patterns for routing prompts	`compass/agents/prompt_builder_conventions_openai.md`
`compass://agents/prompt-builder/conventions-{provider}/{model_family}`	Model-specific conventions addendum (resource template) — returns empty if no addendum exists	`compass/agents/prompt_builder_conventions_{provider}_{model_family}.md`
`compass://agents/input/clarification-skill`	Structured clarification skill — conversational strategy for the input agent	`compass/agents/skills/structured-clarification.md`
`compass://agents/input/defaults`	Default values and override mechanism for optional fields	`compass/agents/user_input/defaults.md`
`compass://agents/data-validation/format-spec`	Data format specification	`compass/agents/data_validation_format.md`
`compass://agents/data-validation/output-spec`	Output format specification	`compass/agents/data_validation_output.md`
`compass://agents/review-agent/guidelines`	Review Agent base system prompt + iterative-phase shared guidelines (concatenated)	`compass/agents/prompts/review_agent_base_system.md` + `review_agent_iterative_base_system.md`
`compass://agents/backend-setup/clarification-skill`	Structured clarification skill for backend setup	`compass/agents/skills/structured-clarification.md`
`compass://agents/backend-setup/taxonomy`	Backend field taxonomy (blocking/non-blocking)	`compass/agents/backend_setup_taxonomy.md`
`compass://agents/backend-setup/defaults`	Backend defaults and pricing resolution	`compass/agents/backend_setup_defaults.md`
`compass://agents/final-report/template`	Markdown skeleton for the final report — section order and placeholders	`compass/agents/prompts/final_report_template.md`

6. Search Algorithm

The Stage-4 search strategy is selected by two module-level constants in compass/agents/prompt_builder/search_ops.py. This repo runs beam search:

_BRANCH_ALGORITHM: AlgorithmType = "beam"
_BRANCH_ALGORITHM_STATE: dict[str, Any] = {"beam_width": 3}

init_search_state reads them when it creates search_state.json. That file is auto-created at Stage 4 entry by _ensure_stage4_search_state (called from _next_action_for_stage_4 in status.py) before _detect_stage_4_phase runs, so cold-start sub-agents always find a real SearchState on disk. advance_step dispatches to advance_round → advance_round_beam.

Component	Where
Models, Pareto dominance, crowding distance, hypervolume	`compass/agents/prompt_builder/search.py`
Round advance (`advance_round_beam`), elite-set update, convergence	`compass/agents/prompt_builder/search_ops.py`
Review Agent overlays (cold-start / iterative / post-cold-start)	`compass/agents/prompts/review_agent_*_overlay_beam.md`
Full algorithm write-up	`docs/algorithm.md`

These two constants are the seam for a different strategy: change them and supply the matching advance_round implementation. AlgorithmType in search.py currently allows only "beam".

7. Directory Guide

Directory	Description
`compass/`	Main Python package: MCP server, agents, eval engine, prompt manager
`compass/mcp/`	MCP server package: `server.py` (app + stage registry), `*_tools.py` (per-stage tools), `resources.py`, `prompts.py`
`compass/agents/`	Root-level modules: `eval_runner.py` (`run_eval`), `routing_context.py`; stage subdirectories below
`compass/agents/user_input/`	User input: input report contract (`report.py`), context/defaults/taxonomy/template resources
`compass/agents/pipeline/`	Pipeline guards (`guards.py`), status detection (`status.py`), dispatch markers and fanout primitives (`dispatch.py`)
`compass/agents/data_validation/`	Data validation: schema checks, format detection, dataset transform, stratified split
`compass/agents/prompt_builder/`	Prompt builder: search state, Pareto ops, holdout filter, best-practices docs
`compass/agents/review/`	Review agent: models, preprocessor, ops
`compass/agents/final_report/`	Final report: models, preprocessor (briefing builder + chart generation)
`compass/agents/prompts/`	Agent system prompts (Markdown) surfaced via MCP
`compass/eval/`	Evaluation engine: controller, backends, metrics, dataset loading, result collection
`outputs/<run_id>/input/`	Pipeline run: validated input report
`outputs/<run_id>/validation/`	Pipeline run: transformed dataset, quality report, routing context
`outputs/<run_id>/analysis/`	Pipeline run: dev/holdout splits
`outputs/<run_id>/prompts/`	Pipeline run: versioned routing prompts (v1.txt, v2.txt, ...)
`outputs/<run_id>/search/`	Pipeline run: search state, pending candidates, candidate archive, `viz.html`
`outputs/<run_id>/search/viz.html`	Self-contained interactive visualization (candidate tree + Pareto scatter + round slider); regenerated after each state mutation by `_try_write_viz` in `search_ops.py`. Nodes are coloured by Pareto elite-set membership.
`outputs/<run_id>/search/child_variants.json`	`ChildVariant[]` — Review Agent output: grouped directives with parent preferences and hypotheses (canonical directive storage; retrieved via `get_child_variants`)
`outputs/<run_id>/rerun_config.json`	Rerun mode marker: `mode`, `source_prompt_version`, `original_backend`, `new_backend` (null until Stage 3 completes)
`outputs/<run_id>/search/search_state_original.json`	Preserved original search state from before rerun initiation
`outputs/<run_id>/eval/`	Pipeline run: dev evaluation results and reports
`outputs/<run_id>/holdout_eval/`	Pipeline run: holdout evaluation results and reports
`outputs/<run_id>/reports/`	Pipeline run: final report (`final_report.md`) and charts (`charts/`)
`backends/`	Backend profiles (YAML, shared across runs)
`tests/`	Test suite (`pytest`)
`tests/scenarios/`	MCP integration test scenarios
`docs/`	Project documentation and specs

8. Installation (External Users)

Add to your project's .mcp.json:

{
  "mcpServers": {
    "compass": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/<owner>/project-compass", "compass"]
    }
  }
}

Then initialize your project:

compass init

This creates outputs/, prompts/, and backends/ with starter files.

All file I/O (outputs/, prompts/, backends/) resolves against the current working directory. To control where files are written, set the working directory when launching the server:

{
  "mcpServers": {
    "compass": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/<owner>/project-compass", "compass"],
      "cwd": "/path/to/your/project"
    }
  }
}

Navigation Menu

Repository navigation

architecture.md

architecture.md

Architecture Map

1. Pipeline Overview

2. Agent Registry

3. Context Dict Reference

4. Shared Models

5. MCP Surface

Tools

Stage-Scoped Tool Filtering

Model Routing

Sub-Agent Guard Pattern

Pipeline guards (dispatch markers and fanout tracking)

Prompts

Resources

6. Search Algorithm

7. Directory Guide

8. Installation (External Users)

FilesExpand file tree

architecture.md

Latest commit

History

architecture.md

File metadata and controls

Architecture Map

1. Pipeline Overview

2. Agent Registry

3. Context Dict Reference

4. Shared Models

5. MCP Surface

Tools

Stage-Scoped Tool Filtering

Model Routing

Sub-Agent Guard Pattern

Pipeline guards (dispatch markers and fanout tracking)

Prompts

Resources

6. Search Algorithm

7. Directory Guide

8. Installation (External Users)