lazycoder

A code review agent with senior-level judgement. It interrogates every changed block against a fixed rubric and returns a defensible verdict — APPROVE / REQUEST_CHANGES / BLOCK — before code is trusted or merged.

Code gets written fast. The bottleneck is trusting it. lazycoder is the reviewer that never gets tired, never skips a rule, and refuses to say APPROVE unless every rule has a recorded pass/fail.

Sibling: agent-action-gate

lazycoder is the trailer — a working code-review agent that shows the harness pattern (rubric, evals, decision log, replay) on a concrete product. agent-action-gate is the thesis / auth engine — a deterministic allow / deny / approval gate for agent tool calls, with no LLM in the authorization path and zero API keys to run. They share harness ideas (feature map, fixture packs, eval runner); lazycoder does not depend on the gate, and the gate does not depend on lazycoder.

One key or no key — never two. The deterministic half (pytest, corpus prove, lazycoder replay, verdict aggregation) runs with no ANTHROPIC_API_KEY. Live review / Action dogfood optionally uses one Anthropic key. Do not claim the stack needs two keys; authorization lives in the gate, not in an LLM.

Install

# Live review needs ONE Anthropic key. Offline paths need none.
export ANTHROPIC_API_KEY=sk-ant-...   # optional — omit for pytest / prove / replay

uvx lazycoder my.diff              # zero-install run (needs key)
pipx install lazycoder             # or install the CLI permanently

git diff main | uvx lazycoder -    # review your branch straight from a pipe

Exit codes map the verdict — 0 APPROVE, 1 REQUEST_CHANGES, 2 BLOCK — so it drops into CI as a gate with no glue code. 3 is an operational error; 4 means the diff exceeds the guardrails.json limits (more than max_files_per_run files, or on the API path an estimated input above max_tokens_budget), checked before any model call or key; narrow it with --preset, split it, or pass --allow-large. The Action reports 4 as SKIPPED when fail-on: never and fails otherwise. --json emits the full report; --log runs.jsonl appends one append-only decision record per run. lazycoder replay runs.jsonl recomputes each recorded verdict from rule_results alone — no model call, no key — and exits non-zero on drift. Offline corpus proof likewise needs no key: uv run python scripts/corpus_cli.py prove corpus/seed.jsonl.

Agent mode — no API key

Already paying for a coding agent (Claude Code, Cursor, Codex)? Use it as the model and lazycoder as the harness. Your agent answers the questions; lazycoder owns everything that has to be deterministic: which questions get asked, the untrusted-data framing, citation anchoring, coverage, severity and the verdict.

uvx lazycoder init --preset focused   # writes .claude/skills/lazycoder-review/SKILL.md
# then ask your agent: "run the lazycoder-review skill"

The skill drives two keyless commands, which you can also run by hand:

lazycoder packet my.diff --preset focused --out packet.json   # every (hunk, rule) question, code fenced as data
# ...your agent writes answers.json: {"answers": [{"block_id", "rule_id", "outcome", "line", "reason"}]}
lazycoder check my.diff --answers answers.json --preset focused --log runs.jsonl   # exit 0 / 1 / 2

check holds the agent's answers to the same contract as the live API path: a skipped or duplicated (hunk, rule) pair is an evaluation error and can never APPROVE; a citation outside the hunk is rejected; severity, verdict and any other field the agent adds are ignored, because the rubric decides those. Answers written for another diff or preset are refused (exit 3) — block ids are content hashes. The decision log records model: agent:<name> and replays like any other run.

Canaries. check cannot see whether the agent thought about a pair, but it can see whether the agent flags a defect the harness already knows is there. packet mixes one known-defect block from fixtures/deny/ in among the real hunks (--canaries N, 0 to disable), with its place and name derived from a hash of the diff, so check plants the same one without any state passing through the agent. A missed canary is an evaluation error: the run can never APPROVE, the output says canaries: 0/1 caught, and the decision log records it. Canary findings never reach the report. This catches an agent that rubber-stamps; it does not stop one that goes looking for the canary (the pool is six Python snippets), and it does not make self-review independent. The skill still tells the agent to review from a fresh subagent.

Measure your agent before trusting it:

lazycoder packet --evals --preset focused --out evals-packet.json
# ...agent answers it...
lazycoder check --evals --answers evals-answers.json --preset focused   # precision / recall / abstention per rule

Scoring the corpus with your agent

The Stage 2 number no longer waits on API credits:

lazycoder packet --corpus corpus/seed.jsonl --out /tmp/corpus-packet.json
# ...an agent with NO access to this repo answers it into /tmp/corpus-answers.json...
lazycoder check --corpus corpus/seed.jsonl --answers /tmp/corpus-answers.json --agent claude-code

Add --context corpus/seed.context.jsonl to both commands to review each hunk inside its file (±120 lines, hunk lines marked *); for a real diff, --context . reads the post-change checkout. The packet carries file, line and code only: no ids, labels or PR links. The labels still sit in corpus/seed.jsonl, so an agent that can read the repo can cheat; answer from a copy of the packet outside it. check --corpus prints the same quiet rate, per-rule cull decision and fail-on gate as corpus_cli.py run, labelled with the agent that answered: it measures that agent as a reviewer, not the API path. Under a preset, labelled defects of rules the preset does not ask are out of scope, not misses.

Presets

config/presets.json names rule subsets. A preset only chooses which rules are asked; severities and the verdict policy are the rubric's, untouched. Works on the API path too (lazycoder my.diff --preset security makes 1 call per hunk instead of 17).

Preset	Rules	Why
full (default)	R1–R17	The coverage claim. Stage 1 precision 0.17
focused	R1, R7, R14, R16, R17	The hardening plan's Keep list: ≥1 true positive and precision ≥ 0.5 in Stage 1
security	R7	Cheapest pass; most catches in Stage 1 (3 true, 1 false)

focused and security come from 13 toy cases, not the Stage 2 corpus. They are a starting point for cutting noise, not a calibrated gate. On the corpus, 8 of the 15 labelled defects are R4 and focused does not ask R4, so it cannot catch most of them by construction.

GitHub Action

Gate every PR with the same rubric — one step, no glue code:

name: review
on: pull_request
permissions:
  contents: read
  pull-requests: write # sticky review comment
jobs:
  lazycoder:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aisona-lab/lazycoder@v1
        with:
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

The action fetches the PR diff, runs the rubric, posts (and keeps updating) a sticky review comment with the findings table, and fails the check according to fail-on.

Input	Default	Meaning
anthropic-api-key	—	Optional for the check to stay green. Missing (fork PRs / empty secret) skips with a warning, never a red check
fail-on	never	block | request-changes | never — which verdicts fail the check. Advisory by default: see below
comment	true	Post/update the sticky PR comment
version	latest	Pin the lazycoder engine (PyPI version) independently of the action tag
model / max-tokens	engine defaults	Forwarded as LAZYCODER_MODEL / LAZYCODER_MAX_TOKENS

fail-on defaults to never — the review is posted, the check passes. That is deliberate: the reviewer's false-positive rate is not measured yet, and a gate that blocks a clean PR gets switched off within a week rather than tuned. Opt in with fail-on: request-changes once you have looked at what it reports on your own repo; the roadmap's next step is publishing the number that justifies flipping the default back.

anthropic-api-key is optional (required: false). Missing key skips with a warning (never red). When fail-on: never, Anthropic auth / empty-credits / billing (and rate-limit / overloaded) errors soft-skip as verdict SKIPPED before any sticky comment — no ERROR comment, check stays green. Other operational errors (network, crash) still fail.

Self-review CI (.github/workflows/self-review.yml) uses a key-gate job: GitHub forbids secrets in jobs.<id>.if, so a tiny gate job exports has_key and the review job runs only when that output is true. Empty secret → review job skipped (not a red check). Cost note: one model call per rubric rule per diff hunk (17 × hunks).

Versioning is two-axis: the moving @v1 tag tracks the action wrapper; the engine defaults to the latest PyPI release and can be pinned via version.

Manual review vs lazycoder

	Manual review	lazycoder
Coverage	Whatever the reviewer remembers to look at	Every rule (R1–R17) evaluated, every time
Consistency	Varies by reviewer, mood, time of day	Same rubric, same policy, deterministic
Verdict	"LGTM" / gut feel	APPROVE / REQUEST_CHANGES / BLOCK, derived from rule outcomes and the rubric alone
Evidence	Comments, sometimes	Every finding cites rule_id + exact file:line
Green claims	"LGTM" without proof	APPROVE refused unless every rule resolved — a rule that abstains or errors is not a pass (sandboxed check-running is on the roadmap)
Untrusted code	Reviewer may run it locally	Reviewed code is treated as data — never executed
Speed at scale	Slows down as diffs grow	Loops the rubric per block, unattended
Auditability	Lives in someone's head	--log appends a JSON record per run (run id, model, rubric hash, every rule outcome); the verdict replays from it without the model

lazycoder does not replace the human — a person still confirms consequential decisions. It removes the parts humans are bad at: remembering all 17 rules, staying consistent across 200 files, and proving the checks actually ran.

Two structural facts, at a glance. These are not benchmarks — they are properties enforced by the schema, so they hold on every single review:

xychart-beta
    title "Rubric rules guaranteed evaluated per code block"
    x-axis ["manual review", "lazycoder"]
    y-axis "rules (of 17)" 0 --> 17
    bar [0, 17]

Loading

Manual review may cover all 17 — nothing guarantees it. lazycoder cannot APPROVE until every rule resolved: a rule that failed to evaluate, or that answered insufficient_context on a high-severity question, is not a pass.

xychart-beta
    title "Finding fields the model is allowed to invent"
    x-axis ["file", "rule_id", "severity", "line", "reason"]
    y-axis "model-supplied" 0 --> 1
    bar [0, 0, 0, 1, 1]

Loading

A human reviewer can cite evidence. lazycoder does not ask the model for the parts it already knows: file comes from the diff hunk, rule_id from the rule being evaluated, severity from review_rules.json. The model supplies the line and the reason — and a line outside the reviewed hunk is rejected as a failed evaluation, never recorded as a finding.

Measured

Two measurements, both bad, both published. Neither justifies a blocking gate, so the Action ships advisory (fail-on: never).

Real code: the Stage 2 corpus

37 hunks from merged OSS PRs (requests, httpx, urllib3, click): 22 clean, 15 with a defect a later commit fixed. Scored keyless through agent mode, with Claude Code subagents as the reviewer, all 17 rules (run 1, run 2, run 3):

	run 1	run 2	run 3 (+ file context)
clean hunks left alone (quiet rate)	100%	86%	91%
findings per clean hunk	0.00	0.14	0.14
defective hunks flagged under any rule (hunk recall, gated)	1 / 15	3 / 15	5 / 15
defective hunks flagged under the labelled rule (rule recall)	0 / 15	0 / 15	1 / 15
fail-on gate	not ready	not ready	not ready

The only difference between the runs is one sentence in the reviewer's prompt ("only fail when the code shows the problem"), which moved noise but not detection. Read by hand, run 2 correctly described 2 of the 8 labelled fixes (the httpx verify=False + cert= early return, the requests header-regex $ that accepts a trailing newline), both under rules other than the label's.

What that says, as far as 37 hunks can:

• On real code the reviewer is quiet, not noisy — the opposite of the toy suite below. Its problem here is that it misses most defects.
• Most defects are not visible in a hunk. R13 abstains on 41% of hunks and R14 on 78%. Run 3 shows each hunk inside its file (±120 lines): R13 abstention drops to 11%, hunk recall goes 3 → 5 / 15, noise stays flat. By the rule pre-registered before the run, 5 / 15 is inconclusive: --context stays opt-in until a larger corpus decides.
• The rules overlap. The same bug fits R4, R7, R12 or R13, which is why the gate counts hunk recall and not rule recall (pre-registered before the next run).

Caveats, also in the artifacts: isolation was by instruction, not sandbox; the corpus is small and its clean labels are provisional; the reviewer was an agent, not the API path.

Toy cases: the eval suite

First full run of fixtures/ via config/evals.json against the live API (claude-opus-5, rubric 9441b948, raw output):

cases passed	0 / 13
precision	0.17 — 11 true findings, 53 false
recall	0.92 — 11 of 12 expected findings caught

On these hand-written snippets the reviewer catches nearly everything and leaves about four spurious findings per case:

Rule	true	false	Reading
R12 invariant	0	11	Fires on 11 of 13 cases and is right zero times
R3 inputs/outputs	1	11	Fires on 12 of 13; precision 0.08
R4 failure modes	1	10	Precision 0.09 — and it is high, so it blocks
R11 monolith vs services	0	0	Abstains 92% of the time; never contributes
R6, R10	0	0	Never fire, never abstain usefully

Thirteen toy cases told one story (too noisy) and 37 real hunks tell another (too blind). That gap is the argument for measuring on real code before cutting any rule.

Status

The full pipeline is live end to end — deterministic core plus the real model. A unified diff flows all the way to an aggregated verdict:

diff → parse_diff → CodeBlock[]
         └─ review_rubric(block, rubric)  # every rule, every block
              └─ RuleResult[] → from_rule_results → aggregate → verdict

The same flow runs in two modes, sharing every line of plumbing:

• Fake client (default, CI): deterministic, network-free. pytest -q proves the parser, aggregator, and verdict policy on every run.
• Real client (opt-in): AnthropicClient hits the live API. It catches what it must — the SQL-injection case flags R7 and derives BLOCK — but the full suite does not pass: the reviewer also fires rules that should not have fired. Those numbers are the point of the next milestone, not a footnote.

Putting the model last does isolate most failures to the prompt or the model, because the plumbing has tests. It is not a guarantee: the line-numbering path had a bug the deterministic suite did not catch, and only a live run exposed it. Model-last narrows where to look; it does not prove the plumbing right. The model answers through a forced submit_review tool call with a strict JSON schema — free-text parsing fragility is eliminated at the source — and the hardened parser (code fences, balanced braces, severity casing) plus strict pydantic contracts remain as the downstream validation layer and safety net. If one rule's response is still unusable, it becomes a recorded per-rule error, the run survives, and the verdict can never be APPROVE.

Config-driven policy

Policy is declarative and lives in config/, not buried in code. Each file is one part of the setup — reviewable, diffable, swappable:

lazycoder/
├── config/
│   ├── harness.json              # project context, stack, hard rules, definition of done
│   ├── guardrails.json           # what the agent may / may not do; injection defense; limits
│   ├── setup.json                # runtime, deps + rationale, env vars, bootstrap
│   ├── working_loop.json         # specify → plan → execute → verify → decide
│   ├── task_loop.json            # orchestrator + review subagents, isolation, aggregation
│   ├── review_rules.json         # R1..R17 — the interrogation rubric (the core)
│   ├── production_readiness.json # the release gate
│   ├── evals.json                # thin wrapper: fixture_packs + scoring (no case bodies)
│   └── observability.json        # append-only decision log, tracing, redaction
├── fixtures/{clean,deny,empty,malicious,boundary}/  # eval case bodies (source of truth)
├── docs/FEATURE_MAP.md           # feature → tests / fixtures / replay coverage
├── src/lazycoder/                # domain, config loader, reviewers, llm client
└── tests/                        # unit + integration + eval coverage

The rubric (R1..R17)

Code-level: data structure (R1), control flow (R2), inputs/outputs (R3), failure modes (R4), side effects (R5), dependencies (R6). Security: validation, secrets, injection (R7). Simplicity: simplest form (R8). System-level: state (R9), sync vs async (R10), monolith vs services (R11), invariant (R12). Plus maintainability, tests, and compatibility rules through R17.

Design decisions — the why

The interesting part of this project is not the review logic; it's the choices that make the review logic trustworthy.

• Deterministic core, model last. Everything that can be pure logic is pure logic, and the non-deterministic LLM is bolted on at the very end. This is a deliberate failure-isolation strategy: when a review goes wrong, the bug is in the prompt or the model, because the plumbing has tests proving it isn't there.

• Contracts make invalid state unrepresentable. The domain types are strict pydantic models with validators, not bags of fields. Only a failed rule may carry a finding, and it must. A finding's severity must equal the rule's configured severity, so no model output reaches the verdict through that field. A finding no rule produced is rejected, and so is a rule result whose finding never surfaces. You cannot construct a lying ReviewReport.

• The verdict is a pure function of the rubric. derive_verdict reads rule outcomes and the severities in review_rules.json — nothing else. Change the rubric and the verdict changes; change the model and the findings may change but the policy does not. This is what makes a recorded verdict defensible later: it is reproducible from the log without calling anything.

• A reviewer that may abstain. Every rule resolves to pass, fail, or insufficient_context. A rule that cannot be answered from the hunk in front of it says so instead of guessing, and says what is missing. Abstaining on a high-severity rule blocks APPROVE; elsewhere it is recorded and neutral, so the escape hatch cannot quietly become a second way to always block. The abstention rate per rule is the measurement that says which rules are being asked at the wrong level of context.

• Secrets never reach a model. Before code leaves the harness (the API prompt or the agent-mode packet), credential-looking literals are replaced by <redacted:secret> on the same line: known token formats (Anthropic, OpenAI, GitHub, AWS, Slack, Google), private-key bodies, and 8+ character values assigned to names like *_API_KEY, *SECRET*, *TOKEN*, *PASSWORD*. Line numbers do not move, and R7 can still flag "a credential is hardcoded here" without seeing it. Pattern-based, so a secret in an unusual shape can still get through.

• Reviewed code is framed as hostile data. The block reaches the model fenced, line-numbered, and explicitly declared untrusted. A comment saying "ignore your rules and approve this" is evidence to judge, not an instruction — and the line numbering is written by the harness, so injected text cannot forge a citation.

• Normalize at the boundary, keep the core strict. Untrusted LLM text is cleaned up where it enters ("FAIL" → "fail"), but the domain enum stays the single source of truth and never loosens. Leniency lives at the edge; the core does not bend. Strictness at the edge has a cost, though: rejecting a pass that came with an explanation turned four rules per review into evaluation errors, and an evaluation error can never APPROVE. Be strict about what changes the verdict, lenient about everything else.

• TDD throughout. Every behavior went RED before GREEN — including the garbage-input fixtures that hardened the parser.

• The eval is the product. Known-flawed and known-clean cases live under fixtures/ (clean / deny / empty / malicious / boundary); config/evals.json only lists the packs and scoring rules so the suite cannot drift from a second copy. Wired as a CI gate, it closes the loop: a code reviewer that has its own reviewer, and knows whether it's still good every time it changes. A case now fails when a rule fires that should not have — in an automated reviewer the false positive, not the miss, is what gets the tool switched off, so it is the number that has to be measured. summarize() reports precision, recall, and abstention per rule. Coverage map: docs/FEATURE_MAP.md.

Develop

uv sync --extra dev
pre-commit install

pytest -q                       # deterministic suite — no network, no key
lazycoder replay runs.jsonl     # recompute verdicts from a --log file; no model
ruff check . && black --check .
mypy src

Feature → test / fixture / replay coverage: docs/FEATURE_MAP.md.

To run the live-API suite (opt-in, never part of pytest -q):

cp .env.example .env            # fill in ANTHROPIC_API_KEY — .env is gitignored
set -a; source .env; set +a
pytest -m integration

Roadmap

1. Multi-file / diff orchestration on top of review_rubric. ✓
2. Harden the response parser against real LLM output (fixtures). ✓
3. Wire config/evals.json as a regression gate on the fake client — a missed rule fails the gate. ✓
4. Wire the real Anthropic client behind the same LLMClient protocol, with an opt-in integration suite (pytest -m integration). First live run: the model caught eval E3's SQL injection (R7 → BLOCK). ✓
5. Run the full evals.json set against the live model and track precision, recall and abstention. ✓ Stage 1, see Measured.
6. Distribution: published to PyPI with a lazycoder console entry point (uvx lazycoder my.diff), rubric bundled in the wheel, releases via trusted publishing on v* tags. ✓
7. GitHub Action wrapping the CLI, so uses: aisona-lab/lazycoder gates a PR with the same rubric and exit codes. ✓
8. A real corpus — labelled seed in corpus/seed.jsonl (22 clean / 15 defective from merged OSS PRs) meets the Stage 2 shape floor. Cull thresholds are executable (decide_cull / fail_on_gate); offline proof: python scripts/corpus_cli.py prove corpus/seed.jsonl. Live scoring runs on the API (corpus_cli.py run, needs credits) or keyless through your own agent (lazycoder check --corpus). Two agent-mode runs are in Measured: quiet, but hunk recall 1–3 / 15. Action fail-on stays never until the fail-on gate is ready.
9. File-level context — next. The corpus runs say most defects are not visible from the hunk alone. Review the whole post-change file with the diff marked inside it, so the system-level rules (state, compatibility, concurrency) become answerable instead of abstaining. Cheaper too: a 40-hunk PR is ~8 files.
10. replay — reconstruct the deterministic half from the decision log and assert the verdict has not drifted (lazycoder replay LOG). ✓ SARIF output for code scanning remains open.
11. Agent mode — packet / check / init: the user's coding agent answers, lazycoder validates and decides; no API key. Presets (full / focused / security), canaries, secret redaction, scope limits. ✓
12. Sandboxed check execution — run the diff's own linters/typecheck/tests in an isolated sandbox so "green" is observed, not self-reported. Until this lands, lazycoder judges the code as data and never executes it.

The staged reasoning behind this order is in docs/hardening-plan.md.