MVP: evidence-driven recursive agentic runtime

[jeremymanning]

Origin and decision

This is a fresh, implementable statement of the idea introduced in #485. It incorporates all eight comments on that issue, including three design reviews, the checked-in simulations in #486, the runnable prototype/measurements, the literature and tools surveys, and the final cross-review synthesis.

The proposal is not a refactor of the legacy recursion or skills modules. It is a new experimental agentic runtime, built alongside the current supported product, that reuses only small verified pieces where their contracts fit.

Problem

Given a goal that is not directly solvable, can Orchestrator:

1. recursively turn it into a bounded, typed plan;
2. prove that claimed leaf operations are actually executable with available capabilities;
3. coordinate logical workers through durable state and messages;
4. review plans and results independently against frozen criteria and evidence;
5. recover from interruption without losing lineage or duplicating completed work; and
6. manage more source material than one model context while keeping every claim traceable?

The MVP must answer that question with measured executions. It is not enough to demonstrate attractive planning transcripts.

Product hypothesis

A small event-sourced workflow kernel plus checked recursive planning can solve bounded, externally verifiable tasks reliably enough to justify a production system, provided that:

• an atomic step is an admitted executable claim, not a planner label;
• corrected recursive branching remains subcritical (m = E[ambiguous children] < 1) on the target task distribution;
• every loop has hard token, cost, node, attempt, time, depth, and fan-out budgets;
• review is evidential, independent by session, and bounded; and
• summaries orient retrieval but never replace addressable source evidence.

Core model

Keep these identities separate and immutable/versioned where applicable:

1. Problem specification — goal, inputs, output schema, constraints, acceptance checks, budgets, and authority.
2. Plan definition/version — a typed control-flow graph.
3. Run / node run / attempt / agent session — execution state and worker assignment. A logical node is not an LLM instance.
4. Artifact/evidence — outputs, source spans, messages, reviews, predictions, tool calls, observations, and context manifests.
5. Capability/version — an executable operation with typed I/O, authority requirements, and qualification evidence.

Use one append-only event log as the source of truth. The execution tree, journal, messages, insight view, capability history, results, and metrics are projections. Large immutable artifacts live in a content-addressed store.

MVP scope

1. New package and typed plan IR

• Add the experiment in a new package alongside the frozen supported orchestrator/ path; keep supported examples green.
• Use a typed, validated machine IR; YAML may be an optional serialization, not the semantic contract.
• Minimum node variants: InvokeCapability, InvokePlan, Decompose, Branch, bounded While, Parallel, AskUser, Return, and Fail.
• Structured control flow only. Any surface goto must compile to validated edges.
• Reject cycles without a machine-checkable bound and reject plans exceeding delegated authority.
• Candidate salvage: core/pipeline.py, core/task.py, core/expressions.py, tools/base.py, checkpointing/state primitives, models/providers/, and supported-example regression discipline.
• Explicitly do not build on tools/pipeline_recursion_tools.py or the legacy skill creator/tester/registry.

2. Single-machine durable execution kernel

• SQLite in WAL mode for events, leases, metadata, budgets, and FTS5; filesystem content-addressed blobs for artifacts.
• Durable parent/child run linkage, attempts, causal event IDs, deterministic projections, and saved context manifests.
• Worker leases, checkpoint-boundary typed message delivery, pause/cancel, and orphan recovery.
• Resume by replay; completed idempotent nodes are not repeated.
• Loud terminal states: completed, failed, blocked, escalated, cancelled, and budget_exhausted. Partial output never appears successful.
• Exact replay for deterministic local capabilities and recorded-response replay for mocked external boundaries. Model calls need lineage, not bit reproducibility.

3. Recursive decomposition with admission control

• A planner emits IR under an explicit fan-out cap and per-child budgets.
• Unknown operations become Decompose nodes; depth/node caps are backstops, not evidence of feasibility.
• Before an InvokeCapability leaf runs, an independent admission check verifies typed I/O compatibility and asks for executable evidence that the named capability can satisfy the step.
• Rejected atomic claims are reclassified for decomposition or escalation.
• Record declared and corrected fan-out/ambiguity, overclaim rate, node count, cost, latency, and outcome on every run.
• Cache positive and negative results within the experiment so an exhausted/failed path is not immediately rediscovered. Budget exhaustion must not be recorded as evidence that a plan is intrinsically bad.

4. Bounded independent review

• Apply review in the MVP to generated plans and final outputs, not every leaf.
• Enforce author-session/reviewer-session separation.
• Review immutable artifact versions against the original problem contract, deterministic checks, and a frozen concern ledger.
• A blocking finding requires a criterion plus reproducible evidence; unsupported concerns become non-blocking risks.
• Findings have stable identity and dispositions: fixed, accepted_risk, invalid, deferred, or superseded.
• Cap rounds/cost/time; unresolved blocking findings escalate. A pass reports residual risk and never claims simply "clean."
• Capture reviewer adjudications so false-positive rate and defect recall can be measured later.

5. Context and coordination substrate

• Store concise operational journal events (intent, decision, observation, assumption, blocker, result), not required private chain-of-thought.
• No global scratchpad mutex. Reads use scoped event snapshots/retrieval; compare-and-swap or short transactions protect only state transitions.
• Implement immutable structural chunks and a cited summary DAG. Every summary points to all children and exact source spans/hashes.
• The guarantee is bounded overview + lossless source addressability + on-demand retrieval, not lossless compression into a context window.
• Use SQLite FTS5 first. Evaluate seeded-needle retrieval and summary routing before adding embeddings.
• Insights in the MVP are run-local, provenance-linked, independently reviewed artifacts projected from the same substrate.

6. Data-driven design and self-measurement

• Plans may carry typed assumptions, predictions, fixtures/generators, invariants, falsifiers, executable experiments, observations, and evaluations.
• Record code hash, dependency lock, command, seed, stdout/stderr, environment, and output artifacts for experiments.
• Deterministic/property-based checks must test likely counterexamples and guard against circular or incomplete oracles.
• Generate contract/status claims from evidence where practical so architecture documentation does not silently rot.

End-to-end demonstrations

The MVP is complete only when all three scenarios run through the same public API and event model:

A. Durable semantics fixture

A root plan contains an atomic node, a child plan, a bounded branch/loop, a pending-node scope-change message, and an intentional failure. Kill the process after recorded progress, resume it, fix a seeded plan defect found by an independent reviewer, and finish with projection and artifact lineage equivalent to an uninterrupted reference run.

B. Real repository repair benchmark

Given a small isolated repository with a failing test and enough irrelevant code to require retrieval, produce and execute a recursive plan that diagnoses the defect, patches it, and returns a passing test plus a trace linking problem -> plan versions -> attempts -> tool calls -> diff -> test evidence -> review. Include multiple seeded defect classes and held-out variants so success is not a scripted demo.

C. Evidence-grounded corpus task

Given a corpus larger than the selected model's working context, answer an externally checkable synthesis question. Every material claim must resolve to source spans. Seed known needles and distractors, and measure source recall, citation support, routing cost, and the effect of summary depth.

Viability gates

Publish raw run artifacts and confidence intervals. Before calling the idea viable:

• At least 50 real decomposition decisions and 30 claimed-atomic steps have independent admission outcomes.
• Corrected m = E[ambiguous children] has an upper confidence bound below 1 on the declared MVP task distribution, or the issue concludes that unrestricted recursive decomposition is not viable for that distribution.
• At least 80% of held-out repair/corpus tasks reach an externally verified result within predeclared budgets; all other outcomes are loud partial/blocked/escalated results.
• Crash/resume and duplicate-delivery fault tests preserve projections and do not repeat completed deterministic effects.
• Seeded critical plan/final-output defects are all detected in the acceptance suite; false positives, review rounds, residual risk, and review cost are reported rather than hidden.
• Every material corpus answer claim resolves to immutable source evidence; seeded-needle retrieval recall is at least 95%.
• No plan can exceed parent authority or its approved node/token/time/cost/fan-out budgets.
• A clean checkout can reproduce the hermetic acceptance suite with one documented command.

Thresholds are preregistered MVP decisions, not universal product claims. If a threshold fails, retain the negative evidence and report which assumption failed rather than widening scope until the demo passes.

Deliverables

• ADR covering identities, event semantics, IR, budgets, authority, review, context guarantee, and explicit non-goals.
• Schemas and validators for problems, plans, events, artifacts, findings, messages, capabilities, budgets, and terminal results.
• Single-machine kernel, CLI/Python entry point, projection/trace export, and replay tooling.
• Planner, atomic admission checker, bounded plan/final review, run-local retrieval/context compiler, and benchmark harness.
• Fault-injection, property, replay, authority, budget, retrieval, and end-to-end tests.
• Measurement report containing corrected branching, atomic overclaim, task success, retrieval, reviewer quality, latency, token/cost, and failure analyses.
• A go/no-go decision for productionization based on the preregistered gates.

Explicitly deferred

• Scale the agentic runtime to distributed workers and messaging #487 — distributed workers, scalable messaging, backpressure, and distributed failure semantics.
• Govern self-authored tools, skills, and reusable plans as capabilities #488 — self-authored capability lifecycle, sandbox enforcement, qualification, repair, and revocation.
• Build governed cross-run memory, retrieval, and solution reuse #489 — governed cross-run memory, solution reuse, hybrid retrieval, contradiction/staleness, retention, and deletion.
• Calibrate and risk-tier independent artifact review #490 — calibrated risk-tiered review beyond MVP plan/final gates.
• Production hardening for unattended and side-effecting agentic runs #491 — unattended/side-effecting production runs, human approval UX, compensation, quotas, SLOs, and disaster recovery.

Open questions requiring research or a product decision

Must be decided before implementation starts

1. Namespace/API: what is the new package name and public entry point? Recommendation: a new experimental namespace in this repository, leaving the current supported path frozen.
2. Plan authoring: machine-generated typed IR or user-authored YAML as the source of truth? Recommendation: typed IR, with YAML only as optional import/export.
3. Benchmark corpus: which concrete repair repositories/defects and which oversized corpus/question become the preregistered acceptance set? The classes above are fixed; exact fixtures must be selected before tuning.
4. Root authority: who sets/approves budgets, and on exhaustion does an attended run ask while an unattended run stops? Recommendation for MVP: attended, ask only at root; children can never widen authority.
5. Provider independence: given the Dartmouth + hosted HuggingFace policy, which model/evidence channel performs atomic admission and review? Recommendation: prioritize executed evidence and deterministic checks; measure same-family bias rather than assuming fresh-session independence solves it.

Research performed inside the MVP

6. What is the cheapest admission test that reduces atomic overclaim enough to make corrected branching subcritical?
7. What are the admission judge's false-positive/false-negative rates, and how should abstention/escalation work?
8. Does corrected branching remain below 1 as the run-local solution library grows under an explicit fan-out cap?
9. Which decomposition prompt/policy produces the best success/cost tradeoff without hiding ambiguity by overclaiming atomicity?
10. How much independent review is beneficial before false positives and cost dominate? What counts as adequate evidence-channel independence?
11. What chunking, summary compression, and FTS query strategy achieves the retrieval target, and when does a summary layer harm routing?
12. Which operational journal fields improve coordination measurably, and how much recent versus retrieved context should each node receive?

Deferred product questions

13. Cross-run/project/user scope and trust policy for insights and reusable solutions (Build governed cross-run memory, retrieval, and solution reuse #489).
14. Containerization, external side-effect classes, compensation, secrets, retention, deletion, and tenant isolation (Govern self-authored tools, skills, and reusable plans as capabilities #488, Build governed cross-run memory, retrieval, and solution reuse #489, Production hardening for unattended and side-effecting agentic runs #491).
15. Required scale/topology, direct-address messaging, and distributed consistency model (Scale the agentic runtime to distributed workers and messaging #487).

References from the originating discussion

• Originating idea and all design discussion: yet another redesign #485
• Executable design simulations: Add the #485 design simulations behind an invariant test #486
• Literature considered there includes MemGPT, RAPTOR, recursive book summarization, LLM self-correction/evaluator-bias/calibration work, and Durable Functions semantics.
• Candidate tools considered there include SQLite WAL/FTS5, Hypothesis, Temporal/Restate semantics, and sandbox providers; the MVP deliberately starts with SQLite and a local executor.