Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 35 additions & 13 deletions src/xagent/core/agent/context/execution.py
Original file line number Diff line number Diff line change
Expand Up @@ -1456,17 +1456,38 @@ def _build_llm_compact_prompt(
"tool calls, tool observations, files or URLs mentioned, "
"decisions made, and open work. Drop duplicated search noise, "
"irrelevant raw payloads, and verbose intermediate text. "
"Preserve exact reusable artifact handles, including file_id "
"values, file: references, markdown file links, URLs, relative "
"paths, absolute paths, output_path, image_path, video_path, "
"artifact filenames, and any other path-like result fields; do "
"not replace machine-usable handles with only descriptive "
"filenames. Clearly separate completed work from remaining work "
"and name the next action needed. "
"Preserve the language of user-facing requests and constraints; "
"if the history is multilingual, keep important details in their "
"original language instead of translating them. "
"Return only the compact summary."
"Preserve exact reusable artifact handles -- file_id values, "
"file: references, markdown links, URLs, paths, output_path, "
"image_path, video_path, artifact filenames, any other "
"path-like field -- never a descriptive filename instead. "
"Preserve, character for character, the values a tool result "
"returned for the records the request points at: their names, "
"the people, organizations or teams they belong to, their "
"identifiers and reference codes, their statuses, dates, "
"counts and totals. Copy such a value or omit it; never "
"paraphrase, substitute, or invent one to complete a pattern. "
"Dropping a raw payload does not license dropping these "
"values; they are not the bulk that instruction covers. "
"Never copy, in whole or in part, a credential, token, key, "
"password, or other authentication material, or personal "
"information the request does not point at; note only that "
"such a value was present and was omitted. If a value is both "
"an identifier the request points at and authentication "
"material, the exclusion wins: omit it. If your budget cannot "
"hold all of this, keep, in this order: first state what is "
"missing and not listed here, with counts; artifact handles; "
"the identifiers and names the request points at; statuses "
"and dates; then the rest. Separate completed work from "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Major, Blocking — pointer] This hunk correctly broadens the compaction summary to preserve names/orgs/statuses/dates, not just numeric figures. But two other prompt strings in this same file were NOT touched and still only mention "figure": the non-LLM compaction summary trailer at execution.py:1245 and _dropped_tool_results_notice at execution.py:1412 (both unchanged since base commit f69c4ec, confirmed outside this PR's diff so not directly commentable here). Both tell the model to treat a missing/unsupported value as unavailable using only "figure"/"value, figure, statistic, table row, quotation, or identifier" — missing the name/org/date/status types this PR fixes everywhere else it touches (grounding.py, auto.py, dag.py, and this hunk). Trigger: any compaction event in a long-running session. Fix: broaden both sentences to the same value-kind scope used here.

"remaining work. Report only what happened and what is "
"missing: never call a dataset complete, fully retrieved, or "
"fully processed unless the history shows every item was "
"returned and every one is still described here; say which "
"parts survive as prose only. Write no instruction to the "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Minor] "Separate completed work from remaining work" (line 1480) sits immediately next to "Write no instruction to the next call about tool use or whether to answer" (this line). The next sentence clarifies "remaining work" means factual gap-reporting, not directives, but a weaker summarizer model could still blur "remaining work" into an implicit next-step instruction. No test can catch this since it requires observing live LLM behavior.

"next call about tool use or whether to answer: that decision "
"is not yours and its tools are unknown to you. Preserve the "
"language of user-facing requests and constraints; keep "
"multilingual details in their original language. Return only "
"the compact summary."
),
},
{
Expand All @@ -1475,8 +1496,9 @@ def _build_llm_compact_prompt(
"Conversation history to compact:\n"
f"{transcript}\n\n"
"Write a concise but complete continuity summary for the next "
"LLM call. The next LLM call should be able to continue without "
"redoing completed tool calls."
"LLM call. Record which tool calls already completed and what "
"they returned, so the next call can judge for itself what "
"still needs doing."
),
},
]
Expand Down
74 changes: 46 additions & 28 deletions src/xagent/core/agent/grounding.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,20 @@
remedy -- ask the user, or finish reporting the gap -- belongs to the calling
pattern, which owns the user-interaction policy this module cannot see.

This is the proposal-A mitigation from issue #1235. It reduces how often
unsourced figures are emitted and makes disclosure the instructed default, but
it cannot repair a session whose evidence compaction already discarded.
This is the proposal-A mitigation from issue #1235. It forbids unsourced values
by default and makes reporting the gap the instructed response, but it cannot
repair a session whose evidence compaction already discarded.
Proposals B (evidence-preserving compaction) and C (provenance tracking and a
data-source gate) remain open.
"""

from __future__ import annotations

VALUE_KINDS = (
"a number, a person or organization name, an identifier or reference "
"code, a date, a status, or a row of a table"
)


def grounding_rule(*, can_call_tools: bool = True) -> str:
"""Return the grounding rule for answer text and, optionally, tool arguments.
Expand All @@ -36,13 +41,14 @@ def grounding_rule(*, can_call_tools: bool = True) -> str:
possible.

Returns:
A prompt fragment forbidding unsupported claims and unsourced
quantitative data, and requiring up-front disclosure of any
illustrative figures. When ``can_call_tools`` is true it also forbids
supplying a fact-carrying tool-call argument that no source provides,
while leaving arguments the model is expected to compose untouched --
except for a fact value written literally inside composed code or
document text, which the sourcing requirement still covers.
A prompt fragment forbidding unsupported claims and unsourced values of
every kind it enumerates, requiring the gap be reported instead, and
confining unsourced content to a current request that explicitly asks
for a template or sample. When ``can_call_tools`` is true it also
forbids supplying a fact-carrying tool-call argument that no source
provides, while leaving arguments the model is expected to compose
untouched -- except for a fact value written literally inside composed
code or document text, which the sourcing requirement still covers.
"""
insufficient_context_rule = (
"If available context is insufficient, say so or use an appropriate "
Expand All @@ -61,17 +67,15 @@ def grounding_rule(*, can_call_tools: bool = True) -> str:
"messages, the retrieved context, or a value an earlier tool result "
"actually returned; never guess one, never substitute a "
"plausible-looking placeholder for one the user has not given, and "
"never carry one over from a different record. This does not restrict "
"values you are expected to compose yourself, such as a search query, "
"code or a command you write to do the work, a message or answer you "
"write to the user, or document text you were asked to produce. A fact "
"value written literally inside such composed code or text is still "
"subject to the sourcing rule above. The rule also does not reach a "
"default or inferred parameter value such as a page size or result "
"limit, which you are expected to decide yourself. Treat "
"a fact-carrying value you cannot source as "
"missing information rather than inventing it, and omit it when the "
"tool allows it to be omitted."
"never carry one over from a different record. The answer you write "
"to the user reaches you as an argument too; it is wording you "
"compose, so this argument standard does not reach it, while the "
"sourcing rule above still governs every fact inside it. This clause "
"does not reach a default or inferred parameter value such as a page "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Minor] tool_argument_rule (starting at line 62) does not use the shared VALUE_KINDS constant introduced elsewhere in this same file/PR; it keeps its own independently-worded list of fact-carrying value types. Minor duplication/drift risk within the same file, same PR, same author who introduced VALUE_KINDS specifically to solve this class of problem.

"size or result limit, which you are expected to decide yourself. "
"Treat a fact-carrying value you cannot source as missing information "
"rather than inventing it, and omit it when the tool allows it to be "
"omitted."
if can_call_tools
else ""
)
Expand All @@ -81,12 +85,26 @@ def grounding_rule(*, can_call_tools: bool = True) -> str:
"statistics, percentages, table rows, or time series) that are not "
"supported by the conversation, retrieved context, or tool results. "
f"{insufficient_context_rule}"
"Never invent figures to fill a gap, and never present invented numbers "
"as real data; produce unsupported figures only when the user explicitly "
"asked for a template, mockup, or illustrative example. Labeling is "
"required either way: if the answer ends up containing any figure that no "
"tool result or provided context supports, whether or not the user asked "
"for one, say so up front, before presenting it, and state that those "
"figures are illustrative placeholders not drawn from any data source."
f"Never fill a gap with an invented value, whether it is {VALUE_KINDS}: "
"when nothing in this conversation, the provided context, or a tool "
"result supports a value the answer needs, leave that value out and "
"say plainly that it is missing, rather than supplying one that "
"looks right. This does not restrict the wording you compose -- how "
"you phrase your reply, a search query, code or a command you write "
"to do the work, or document text you were asked to produce -- it "
"restricts every fact asserted inside that wording. A fact value "
"written literally inside such composed code or text is still "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Minor] "the text you compose is yours; {VALUE_KINDS} that you place inside it is not" reads, on first parse, as the trailing relative clause ("that you place inside it") binding only to the last list item ("a row of a table"), not the full six-item disjunction. Recoverable from context but worth a comma/rephrase for readability.

"subject to the sourcing rule above: the text you compose is yours; "
f"{VALUE_KINDS} that you place inside it is not. The only case in "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Minor] The template/sample exception is scoped to "in the answer" (stated twice, lines 98-105). When a user explicitly asks for template content that must go into a tool argument rather than the chat answer (e.g. "write a sample invoice and save it to a file"), the exception's literal wording doesn't reach that content, and no other clause broadens it. This is a newly-sharpened gap — the base wording wasn't scoped to "the answer". No test covers this scenario. Suggest extending the exception to cover composed tool-argument content too.

"which content that no source supports may appear in the answer is a "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Minor] The template-exception trigger phrase "a template, a sample, or content that is not meant to be real" — "a sample" alone is ambiguous (could mean "an example populated with real data", not just mockup content); "explicitly asks" constrains that the user asked, not what they meant by "sample". No test covers this ambiguous-intent edge case.

"current user request that explicitly asks you to write a template, "
"a sample, or content that is not meant to be real; in that case, "
"before any of that content appears in the answer, state that the "
"request asked for content that is not real and that none of it "
"comes from a data source, and keep such content to what the request "
"asked for. Outside that case a caveat does not make an invented "
"value acceptable: if you find yourself about to add a note "
"explaining that some values are not real, remove those values and "
"report the gap instead."
f"{tool_argument_rule}"
)
9 changes: 5 additions & 4 deletions src/xagent/core/agent/pattern/auto/auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
build_load_skill_tool,
)
from ...frame import ExecutionFrame, ExecutionSnapshot, ExecutionStatus
from ...grounding import grounding_rule
from ...grounding import VALUE_KINDS, grounding_rule
from ...language import (
final_answer_language_rule,
reset_metadata_output_language,
Expand Down Expand Up @@ -1281,9 +1281,10 @@ def _decision_prompt(
"answer field in the same tool call. Put action before answer in the "
"tool arguments. "
f"When writing that answer field: {grounding_rule(can_call_tools=False)} "
"If the answer would need such unsupported specifics, set "
"existing_context_sufficient=false and choose react so the agent can "
"verify them with tools.\n\n"
"If the answer would need any value the rule above forbids you to "
f"supply -- {VALUE_KINDS} that no source here supports -- set "
"existing_context_sufficient=false and choose react, so the agent "
"can obtain it with tools.\n\n"
f"{final_deliverable_file_reference_instructions(can_lookup=False)}\n\n"
"You must classify whether "
"the latest request requires current or external facts, and whether "
Expand Down
5 changes: 2 additions & 3 deletions src/xagent/core/agent/pattern/dag/dag.py
Original file line number Diff line number Diff line change
Expand Up @@ -1569,9 +1569,8 @@ def _completion_assessment_messages(self, context: Any) -> list[dict[str, Any]]:
"over from candidate_output or step_results: "
f"{grounding_rule(can_call_tools=False)}\n\n"
f"{final_deliverable_file_reference_instructions(can_lookup=False)}\n\n"
"If the answer presents any figure as an illustrative "
"placeholder because no step produced the underlying data, "
"name that unsourced data in reason even when you choose "
"If the answer leaves out a value because no step produced "
"it, name that missing data in reason even when you choose "
"status=completed. "
f"{final_answer_language_rule(subject='output language policy')}"
),
Expand Down
19 changes: 17 additions & 2 deletions tests/core/agent/test_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
ReActPattern,
)
from xagent.core.agent.context.enrichment import MEMORY_CONTEXT_METADATA_KEY
from xagent.core.agent.grounding import VALUE_KINDS
from xagent.core.agent.language import (
OUTPUT_LANGUAGE_METADATA_KEY,
OUTPUT_LANGUAGE_SOURCE_METADATA_KEY,
Expand Down Expand Up @@ -1052,12 +1053,26 @@ async def test_auto_decision_prompt_includes_grounding_rule() -> None:
assert result["success"] is True
decision_prompt = llm.calls[0]["messages"][-1]["content"]
assert "quantitative data" in decision_prompt
assert "illustrative placeholders" in decision_prompt
assert (
"a current user request that explicitly asks you to write a template"
in decision_prompt
)
assert "invented values" in decision_prompt
assert decision_prompt.count("## FINAL DELIVERABLE FILE REFERENCES") == 1
assert decision_prompt.index(
"If the answer would need such unsupported specifics"
"If the answer would need any value the rule above forbids"
) < decision_prompt.index("## FINAL DELIVERABLE FILE REFERENCES")
# The routing remedy stays specific to auto's own decision, so it is
# worded independently of the shared rule's neutral gap-reporting text.
assert (
"set existing_context_sufficient=false and choose react, so the agent "
"can obtain it with tools" in decision_prompt
)
# The value kinds are not auto's own wording: the sibling sentence
# interpolates the shared constant, so this pins the reference rather
# than restating the list.
assert f"{VALUE_KINDS} that no source here supports" in decision_prompt
assert "such unsupported specifics" not in decision_prompt
assert "get_workspace_output_files" not in decision_prompt
assert "You must classify whether" in decision_prompt
assert "You must also classify whether" not in decision_prompt
Expand Down
Loading