Skip to content

fix(agent): stop turn early on repeated identical tool failure - #3312

Open
lucapette wants to merge 1 commit into
sipeed:mainfrom
lucapette:fix/repeated-tool-failure-circuit-breaker
Open

fix(agent): stop turn early on repeated identical tool failure#3312
lucapette wants to merge 1 commit into
sipeed:mainfrom
lucapette:fix/repeated-tool-failure-circuit-breaker

Conversation

@lucapette

Copy link
Copy Markdown

📝 Description

Fix the "stuck / never answers" behavior seen when a tool fails with the same error on every call (e.g. git without credentials, or any command blocked by the shell safety guard). Previously the agent loop kept re-calling the LLM and re-executing the identical failing tool until max_tool_iterations — with zero user feedback in between — so messages over channels like Telegram appeared to be silently ignored.

This PR adds a circuit breaker: consecutive identical (tool, error) failures are tracked in turnState.recordToolExecution, and when the same tool+error repeats repeatedFailureThreshold (3) times, the turn stops early with a clear, user-visible message explaining the repeated failure.

🗣️ Type of Change

  • 🐞 Bug fix (non-breaking change which fixes an issue)

🤖 AI Code Generation

  • 🛠️ Mostly AI-generated (AI draft, Human verified/modified)

🔗 Related Issue

Fixes #3311

📚 Technical Context

  • Reasoning: pkg/agent/pipeline_execute.go:697 feeds every tool result (including errors) back to the LLM as a prompt message, letting the model retry indefinitely. pkg/tools/shell.go:337 returns safety-guard failures as ErrorResult (an error, not a user-visible message). With no circuit breaker the loop runs to MaxIterations, and if the service restarts first the turn dies silently. recordToolExecution already records per-tool success/failure history, so it is the natural place to detect consecutive identical failures.

🧪 Test Environment

  • Hardware: Apple M1 Max MacBook (build + unit tests)
  • OS: macOS (darwin/arm64)
  • Model/Provider: deepseek-v4-flash via DeepSeek (production repro, v0.3.1 on Linux Mint / Telegram)
  • Channels: Telegram

📸 Evidence

Click to view Logs

Production repro (v0.3.1) — message 918/turn-32 spun through 39 tool iterations of identical failures, no user output:

19:39:00 ERR tool ... > Tool execution failed error="Command blocked by safety guard (dangerous pattern detected)" duration=0 tool=exec
... (repeated ~30x, for_user_len=0 on every event) ...

Before fixgo test -run TestRepeatedIdenticalToolFailure:

BUG: repeated identical tool failure looped to max_tool_iterations: tool executed 5 times (cap=5), no early stop
BUG: user got generic max_tool_iterations message instead of a message explaining the repeated failure

After fix — same test:

INF agent turn_coord.go:240 > Stopping turn: repeated identical tool failure failures=3 tool=mock_fail
PASS

☑️ Checklist

  • My code/docs follow the style of this project.
  • I have performed a self-review of my own changes.
  • I have updated the documentation accordingly.
  • I have added tests that prove my fix is effective.
  • All new and existing tests pass (go test ./pkg/agent/ -count=1 → ok).

When a tool fails with the same error on every call (e.g. git without
credentials, or a safety-guard-blocked command), the agent loop keeps
re-calling the LLM and re-executing the same broken tool until
max_tool_iterations, with zero user feedback. Messages sent over channels
like Telegram then appear to be silently ignored.

Add a circuit breaker: track consecutive identical (tool, error) failures
in turnState.recordToolExecution, and when the same tool+error repeats
repeatedFailureThreshold (3) times, stop the turn early with a clear,
user-visible message explaining the repeated failure instead of spinning
to the cap.

Fixes the root cause behind unanswered messages; self-reproducing test
included (fails on v0.3.1, passes with this change).

Ref: sipeed#3311
@CLAassistant

CLAassistant commented Aug 2, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions

Copy link
Copy Markdown

This PR has had no activity for 7 days and has been marked as stale. If you are still working on it, please push an update or leave a comment; otherwise it will be closed automatically in 7 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Repeated identical tool failure loops silently to max_tool_iterations — user never gets an answer

2 participants