Most of our investigation tools run fine but their output disappears before the agent reads it. Audit findings.

[hamzzaaamalik]

Hey team. I went down a rabbit hole on something and want to share what I found.

The short version

When the agent investigates an alert it runs a bunch of tools. Each tool fetches real data. Then there is a step that takes all that tool output and hands it to the diagnose stage so the agent can write the root cause.

The handoff step has a quiet bug. If a tool is not in a specific list, its output gets thrown away. The tool ran. The data was real. It just never reaches the diagnose step.

I checked every tool in the codebase. Out of 121 investigation tools, only 39 are in the list. The other 82 were not. After looking at each one carefully I classified them. 14 of those are tools that should not be in the list (they are stubs or they do mutations, not data fetches). That leaves 68 tools that run, return useful data, and have that data silently dropped before the agent ever sees it.

Why this matters

When someone asks the agent about a Kafka consumer lag spike today, the Kafka tool runs, returns the lag and group info, and then the agent writes a root cause without ever seeing that data. Same for MySQL, MongoDB, Sentry, Opsgenie, Kafka, RabbitMQ, GitLab, and a long list of others.

This is a real quality issue. The agent looks confident in its answer. The user has no way to know that a relevant tool ran and that its output was thrown out before reasoning started.

Why this happened

I do not think anyone caused this on purpose. The list is in one file and adding a new tool to it is easy to forget when you are focused on the tool itself. The tool passes its tests. The agent looks like it works. The bug only shows up when you trace what data actually reaches the diagnose step.

This same pattern was already caught and fixed for EKS in PR #584. Issues #670 and #672 flag a similar gap for healthy short circuit logic. So the team has seen the shape of this problem before. I just wanted to find out how widespread it is. The answer is wider than the past fixes covered.

What I think we should do

In order:

1. Add a small test that prevents this from happening again. A single test that walks every registered tool and asserts it is either in the mapping list or in a deliberate skip list. If a future contributor adds a new tool and forgets the mapping the test fails the PR. This is the highest leverage thing and it is a tiny change.

2. Add the missing mappings in small per family batches. The 68 tools group cleanly by provider. Database family, error tracking, messaging, observability, version control, and so on. Each family PR is self contained and reviewable in one sitting.

3. Do not open a single 68 tool mega PR. Too big to review and presumes priorities the team has not signed off on.

I am happy to do step 1 myself and a couple of the per family batches. The rest can be picked up by anyone since the pattern is mechanical once the test is in place.

Three questions for maintainers

1. Does the enforcement test sound like the right first PR
2. Which family of tools would you most want to see fixed first. The audit suggests databases (32 tools, biggest single block) but that is a guess about what users hit most
3. There are 14 tools I excluded from the list because they look like mutations or stubs. Are there any in particular that should actually be included

If anyone wants the full audit with the per tool table and the script that produced the numbers I have it locally and can share.

Thanks. Curious to hear thoughts.

CC @davincios @VaibhavUpreti

[hamzzaaamalik]

[
Uploading Evidence mapper audit.pdf…
](url)

[YauhenBichel]

Hi @hamzzaaamalik I created stories for this discussion #5519
Thank you!