JIT Context OS: Running autonomous coding loops on local 9B models with sub-3ms RYOW & zero context bloat (EXP-009)

[wojciechwiesner]

Background & The Agent Context Bottleneck

When running autonomous multi-turn agents (especially on local LLMs like Qwen or DeepSeek), long-running sessions inevitably suffer from the "Haystack Tax":

1. Terminal outputs, code execution logs, and file contents accumulate in the conversation window.
2. The model suffers from Lost-in-the-Middle attention degradation.
3. The agent begins trusting its own speculative or failed outputs from prior turns as factual truth (epistemic self-poisoning).

What is JIT Context OS?

We built and open-sourced JIT-Context OS (CERN Zenodo DOI: 10.5281/zenodo.22649542 · GitHub: wojciechwiesner/jit-context).

Instead of accumulating unbounded conversation history or relying on slow external vector DBs, JIT Context introduces a 3-tier cascade governed by 10 Epistemic Invariants:

• L0 Hot-Path (<3ms SQLite WAL): Fast on-disk Read-Your-Own-Writes (RYOW) capturing confirmed execution state.
• Epistemic Authority Weights: Assistant output is assigned authority 0.0 (speculations cannot pollute ground truth). Only deterministic tool executions (exit_code: 0 or verified readbacks) earn authority 1.0.
• Calibrated Context Capsule (<1.8k tokens): Rather than passing the full chat history, each model call receives a compact working set (Active Invariants + Dev Runtime + Runtime Proofs + Task Goal) alongside the immediate previous tool result.

---

Empirical Proof (EXP-009): Local 9B on Apple Silicon vs Google Gemini 3.8 Flash

To test whether local models can handle real multi-module autonomous engineering when context is kept lean, we ran a head-to-head duel on a task fixing 4 distinct root causes across 3 interconnected Python modules (event_pipeline.py, retry_policy.py, storage.py), evaluated by a blind pytest suite:

Competitor	Hardware & Runtime	Total Turns	Wall-Clock Time	Pytest Verification	API / Hardware Cost	Privacy Guarantee
🥇 Local Qwen 3.8 9B + JIT	Apple Mac Mini M2 Pro (Ollama 32k)	4 turns	~2 min (126s)	4/4 PASSED (exit 0)	$0.00	100% Local / Zero Data Leak
🥈 Gemini 3.8 Flash + JIT	Google Cloud Frontier API	9 turns	17.7s	4/4 PASSED (exit 0)	Paid Cloud API	Cloud API payload
🥉 Gemini 3.8 Flash (No JIT / Chat)	Google Cloud Frontier API (Haystack)	10 turns	34.5s	❌ 0/4 FAILED (lost in tests/)	Paid Cloud API	Cloud API payload

Turn Breakdown for Local Qwen 3.8:

• Turn 1 (Parallel Discovery): Dispatched 4 parallel read_file calls in one shot.
• Turn 2 (Surgical Multi-File Fix): Emitted 3 parallel write_file calls addressing all 4 root causes simultaneously.
• Turn 4 (Verification): Executed run_tests() → 4/4 passed (100% PASS, exit code 0) on the first attempt.
  Meanwhile, Gemini 3.8 Flash without JIT got caught in a regression loop, modifying test files instead of source code.

---

Mapping to Agent Zero

This pattern maps directly into Agent Zero's extension architecture (/a0/extensions/python/):

• tool_execute_after / hist_add_tool_result: Intercept heavy tool outputs, record deterministic proofs to SQLite WAL, and keep raw logs out of active memory.
• before_main_llm_call: Inject a compiled working-set capsule (<1.8k tokens) instead of the accumulated monologue.
• system_prompt: Enforce strict authority gating (I1–I10).

We're exploring packaging this as a drop-in extension for Agent Zero. Would love to hear thoughts from the Agent Zero community on context pruning and local model orchestration!

Repo: https://github.com/wojciechwiesner/jit-context
CERN Zenodo: https://doi.org/10.5281/zenodo.22649542