Skip to content

fix(agent): hand a forced turn back the tools compaction emptied, once - #2148

Open
AlexLiu190625 wants to merge 7 commits into
xorbitsai:mainfrom
AlexLiu190625:fix/react-forced-answer-recovery
Open

fix(agent): hand a forced turn back the tools compaction emptied, once#2148
AlexLiu190625 wants to merge 7 commits into
xorbitsai:mainfrom
AlexLiu190625:fix/react-forced-answer-recovery

Conversation

@AlexLiu190625

@AlexLiu190625 AlexLiu190625 commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stacked on fix/react-forced-answer-honest-instruction (#2147); merge that one first. Until that PR merges, this diff also shows its commits.
Saying that the evidence is gone is the fallback, not the goal. When this turn's
compaction destroys the observations a forced turn was going to answer from, and
the run can still afford it, this change undoes the forcing for exactly one turn
and hands back only the reading tools whose results went missing, plus
final_answer so the model can still finish. The turn after that one is always
the forced answer turn: if the re-fetch brought the values back it gets the
ordinary instruction, and if it did not it gets the honest one. A tool-protocol
repair inside the same turn now rebuilds the request from that turn's own tool
set and keeps the honest wording instead of restoring the sentence the first call
had already replaced.

Behavior changes

  • A forced turn whose own compaction dropped tool results has its forcing undone
    for one turn, and receives the dropped reading tools plus final_answer.
  • Tools that lost nothing stay out of that set, and so do the two control tools
    that contact the user rather than read anything.
  • A system message on that turn names the missing observations, states that
    fetching them again outranks the standing advice against repeating tool work,
    limits re-runs to reading tools, and withholds the final answer until the
    values are back or reported missing.
  • The turn immediately after a recovery turn is always forced back to
    final_answer, with the instruction chosen by whether the evidence actually
    returned.
  • Whether the re-fetch worked is decided from the turn's own tool call ids, not
    from tool names. The dropped observation was itself a successful call of one of
    those names and its ledger entry outlives the compaction, so a name match would
    always claim the evidence came back.
  • A same-turn tool-protocol repair now receives the turn's own tool set and the
    fact that evidence was dropped. Ordinary and forced turns are unchanged: the
    repair still offers the run's whole tool set and its wording is byte-identical.
    A turn that narrowed its own tools gets that narrowed set widened only by the
    tools it is still entitled to, and the "re-decide with the complete tool set"
    line becomes accurate for it.
  • A recovery turn clears the persisted evidence marker together with the loop-local flag, so a pause during the recovery turn resumes into a recovery turn rather than a forced answer; test_a_recovery_turn_resumes_into_a_recovery_turn fails without that clearing line.

Bounds and residual risk

  • At most one recovery per pattern instance. The spent count travels in the
    checkpoint, so a pause and resume cannot buy a second one.
  • Never when fewer than two iterations remain, one to re-fetch and one to answer,
    so recovering can never cost the run its answer; a run with room for a single
    call therefore never recovers.
  • Never for a forcing that means "stop and report blocked", and never for the
    turn a previous recovery already produced, which is what stops recovery from
    chaining into itself.
  • Residual risk: the restored set is chosen by which results were dropped, so it
    can contain a tool that writes as well as reads. The limit to reading is stated
    in the prompt, not enforced by the tool layer. Marking tools read-only in the
    tool protocol, so this set can be filtered rather than described, is the
    follow-up that closes it.
  • Keeping the repair set wider than a narrowed turn's own set is deliberate: a
    model that needs some other tool to reach the dropped values would otherwise
    name it twice and fail the run outright.

Not in this PR

  • Any change to the honest instruction itself or to the three checkpoint keys,
    which land in the branch this one stacks on.
  • Read-only enforcement in the tool protocol.
  • Rewriting the guidance the repeated-tool decision writes into the context; the
    strict xfail test carried by the first branch still records that gap.

Verification

  • tests/core/agent/test_react.py, tests/core/agent/test_grounding.py and
    tests/core/agent/test_react_forced_answer_compaction.py: 314 passed,
    1 xfailed. This branch adds 38 cases to that file on top of the branch it
    stacks on, which alone runs 39 passed and 1 xfailed there.
  • Mutation checks on the production file, each reverted immediately: making the
    re-fetch verdict match by tool name instead of the turn's own call ids, giving
    the recovery turn the run's whole tool set, dropping the recovery reason from
    the set exempt from recovery, clearing the follow-up marker as soon as it is
    read, folding every control tool into the restored set, rebuilding the repair
    request without the dropped-evidence fact, and rebuilding it from the run's
    whole tool set. Each turned red, caught by
    test_an_older_successful_call_does_not_pass_for_a_recovered_one,
    test_recovery_hands_back_the_tools_whose_results_were_dropped,
    test_the_recovery_budget_is_spent_once_per_pattern, the recovery-chain
    tests, test_the_retry_that_produces_the_answer_is_honest_too and
    test_the_protocol_retry_uses_this_turns_own_tool_set.
  • test_an_evidence_loss_still_ends_the_run_with_an_answer walks the outcome
    matrix and asserts every one of them ends the run with an answer and never at
    the iteration ceiling.
  • pre-commit run --files on both changed files: ruff check, ruff format, mypy,
    isort, codespell and the whitespace hooks all pass.

Part of #2146

A ReAct turn forced back to final_answer alone can lose the tool
observations that answer was meant to rest on, because the same turn's
compaction removes them after the decision to answer was already made.
The turn still asked for an answer "using the accumulated conversation
and tool results", which on that turn is a request to invent them.

The forced instruction now has a second form. When this turn's
compaction destroyed tool evidence, it names the removed observations,
forbids reconstructing, estimating or illustrating a removed value, and
withdraws outcome=completed for the turn. The ordinary wording is
replaced rather than appended to, so the two cannot contradict each
other.

Choosing between the two forms needs to know why the turn was forced, so
every site that forces one now records its reason alongside the flag,
and the reason, the recovery budget and the follow-up marker all travel
through checkpoints. Their reload defaults differ on purpose: an
unreadable reason or marker does less, while a missing recovery counter
reads as unspent so runs checkpointed before it existed keep the path.

Refusing to restore tools is recorded in pattern state as well, so an
interrupt between the refusal and the answer resumes into a turn that
still remembers to be honest.
@XprobeBot XprobeBot added the bug Something isn't working label Sep 5, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a mechanism to handle forced-answer turns in the ReAct reasoning loop when the underlying tool evidence has been removed by context compaction. It defines explicit reasons for forced answers, tracks compaction recovery attempts within a budget, and either temporarily restores the dropped tools for a single recovery turn or instructs the LLM to honestly report the missing evidence rather than hallucinating or estimating it. Additionally, it updates state serialization to support these new fields across checkpoints and adds a comprehensive test suite to validate the recovery and fallback behaviors. As there are no review comments, no feedback is provided.

…-dropped turn

The forced-answer gate decided, per turn, whether to send the honest
"evidence was dropped" instruction via a loop-local variable, but only
persisted that decision to the recovery-followup marker on the branch
where recovery was declined. A turn where the gate found dropped
evidence but recovery had not yet been declined left the marker unset,
so an interrupt between the decision and the LLM call resumed into a
forced turn that had forgotten the evidence was ever dropped.

The marker write now happens at the gate itself, alongside the
loop-local it mirrors, so both halves of the same decision are set
together regardless of which branch runs afterward. The decline branch
no longer duplicates the write.
When compaction destroys the observations a forced turn was going to
answer from, saying so is the fallback, not the goal. If the run can
still afford it, the better move is to undo the forcing for one turn and
hand back exactly the tools whose results went missing, plus
final_answer so the model can still finish.

Handing tools back is bounded on every side. It happens at most once per
pattern, never for a forcing that means "stop and report blocked", never
for the turn a previous recovery already produced, and never without two
iterations left -- one to re-fetch and one to answer -- so recovering can
never cost the run its answer. Tools that lost nothing stay out, and so
do the two that talk to the user rather than read anything.

A system message names the missing observations, states that fetching
them again outranks the standing advice against repeating tool work,
limits re-runs to reading tools, and withholds the final answer until
the values are back or reported missing. Its tool-name list mirrors the
bounds the compaction notice already applies.

Whether the re-fetch worked is read from the turn's own tool call ids
rather than tool names. The dropped observation was itself a successful
call of one of those names and its ledger entry outlives the compaction,
so a name match would always claim the evidence came back. The verdict
decides which instruction the following turn gets.
A tool-protocol repair rebuilds the whole request from scratch, and on a
turn whose evidence was destroyed it is the call that actually produces
the user's answer. It was rebuilding the prompt without knowing that,
handing back the same "answer from the accumulated results" sentence the
first call had replaced, so the repair undid the honest wording.

The repair path now receives that fact along with the tool set the turn
is entitled to. Ordinary and forced turns are unchanged: the repair
still hands back the run's whole tool set, and its wording is
byte-identical. A turn that narrowed its own tools gets that set minus
the two tools that contact the user rather than read anything, and the
"re-decide with the complete tool set" line becomes an accurate one,
since the repair set is wider than what the turn first offered but still
not everything the run has.

Keeping the repair set wider than the narrowed one is deliberate. A
model that needs some other tool to reach the dropped values would
otherwise name it twice and fail the run outright, which the narrowed
turn made reachable and no earlier behaviour did.
A recovery turn undoes the forcing and hands back the tools whose
observations compaction dropped. The marker that carries the same
decision across a checkpoint was left behind, so a pause inside a
recovery turn resumed into a forced answer that reported the evidence
unavailable instead of continuing to fetch it. Both halves of the
decision are now undone together, before the recovery checkpoint that
would otherwise persist the stale marker.

The resume case for a recovery turn had no coverage, so the test that
paused a cleared turn now asserts the recovery half rather than the
refusal it used to assert on a build that could not recover.
@AlexLiu190625
AlexLiu190625 force-pushed the fix/react-forced-answer-recovery branch from d9cb40f to af4198f Compare September 6, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants