-
Notifications
You must be signed in to change notification settings - Fork 61
fix(agent): stop the grounding rule and compaction summaries from licensing fabricated answers #2149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
fix(agent): stop the grounding rule and compaction summaries from licensing fabricated answers #2149
Changes from all commits
1d8609f
5b0376e
7bf4ea9
ab52bef
cad2a02
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1456,17 +1456,38 @@ def _build_llm_compact_prompt( | |
| "tool calls, tool observations, files or URLs mentioned, " | ||
| "decisions made, and open work. Drop duplicated search noise, " | ||
| "irrelevant raw payloads, and verbose intermediate text. " | ||
| "Preserve exact reusable artifact handles, including file_id " | ||
| "values, file: references, markdown file links, URLs, relative " | ||
| "paths, absolute paths, output_path, image_path, video_path, " | ||
| "artifact filenames, and any other path-like result fields; do " | ||
| "not replace machine-usable handles with only descriptive " | ||
| "filenames. Clearly separate completed work from remaining work " | ||
| "and name the next action needed. " | ||
| "Preserve the language of user-facing requests and constraints; " | ||
| "if the history is multilingual, keep important details in their " | ||
| "original language instead of translating them. " | ||
| "Return only the compact summary." | ||
| "Preserve exact reusable artifact handles -- file_id values, " | ||
| "file: references, markdown links, URLs, paths, output_path, " | ||
| "image_path, video_path, artifact filenames, any other " | ||
| "path-like field -- never a descriptive filename instead. " | ||
| "Preserve, character for character, the values a tool result " | ||
| "returned for the records the request points at: their names, " | ||
| "the people, organizations or teams they belong to, their " | ||
| "identifiers and reference codes, their statuses, dates, " | ||
| "counts and totals. Copy such a value or omit it; never " | ||
| "paraphrase, substitute, or invent one to complete a pattern. " | ||
| "Dropping a raw payload does not license dropping these " | ||
| "values; they are not the bulk that instruction covers. " | ||
| "Never copy, in whole or in part, a credential, token, key, " | ||
| "password, or other authentication material, or personal " | ||
| "information the request does not point at; note only that " | ||
| "such a value was present and was omitted. If a value is both " | ||
| "an identifier the request points at and authentication " | ||
| "material, the exclusion wins: omit it. If your budget cannot " | ||
| "hold all of this, keep, in this order: first state what is " | ||
| "missing and not listed here, with counts; artifact handles; " | ||
| "the identifiers and names the request points at; statuses " | ||
| "and dates; then the rest. Separate completed work from " | ||
| "remaining work. Report only what happened and what is " | ||
| "missing: never call a dataset complete, fully retrieved, or " | ||
| "fully processed unless the history shows every item was " | ||
| "returned and every one is still described here; say which " | ||
| "parts survive as prose only. Write no instruction to the " | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Minor] "Separate completed work from remaining work" (line 1480) sits immediately next to "Write no instruction to the next call about tool use or whether to answer" (this line). The next sentence clarifies "remaining work" means factual gap-reporting, not directives, but a weaker summarizer model could still blur "remaining work" into an implicit next-step instruction. No test can catch this since it requires observing live LLM behavior. |
||
| "next call about tool use or whether to answer: that decision " | ||
| "is not yours and its tools are unknown to you. Preserve the " | ||
| "language of user-facing requests and constraints; keep " | ||
| "multilingual details in their original language. Return only " | ||
| "the compact summary." | ||
| ), | ||
| }, | ||
| { | ||
|
|
@@ -1475,8 +1496,9 @@ def _build_llm_compact_prompt( | |
| "Conversation history to compact:\n" | ||
| f"{transcript}\n\n" | ||
| "Write a concise but complete continuity summary for the next " | ||
| "LLM call. The next LLM call should be able to continue without " | ||
| "redoing completed tool calls." | ||
| "LLM call. Record which tool calls already completed and what " | ||
| "they returned, so the next call can judge for itself what " | ||
| "still needs doing." | ||
| ), | ||
| }, | ||
| ] | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,15 +13,20 @@ | |
| remedy -- ask the user, or finish reporting the gap -- belongs to the calling | ||
| pattern, which owns the user-interaction policy this module cannot see. | ||
|
|
||
| This is the proposal-A mitigation from issue #1235. It reduces how often | ||
| unsourced figures are emitted and makes disclosure the instructed default, but | ||
| it cannot repair a session whose evidence compaction already discarded. | ||
| This is the proposal-A mitigation from issue #1235. It forbids unsourced values | ||
| by default and makes reporting the gap the instructed response, but it cannot | ||
| repair a session whose evidence compaction already discarded. | ||
| Proposals B (evidence-preserving compaction) and C (provenance tracking and a | ||
| data-source gate) remain open. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| VALUE_KINDS = ( | ||
| "a number, a person or organization name, an identifier or reference " | ||
| "code, a date, a status, or a row of a table" | ||
| ) | ||
|
|
||
|
|
||
| def grounding_rule(*, can_call_tools: bool = True) -> str: | ||
| """Return the grounding rule for answer text and, optionally, tool arguments. | ||
|
|
@@ -36,13 +41,14 @@ def grounding_rule(*, can_call_tools: bool = True) -> str: | |
| possible. | ||
|
|
||
| Returns: | ||
| A prompt fragment forbidding unsupported claims and unsourced | ||
| quantitative data, and requiring up-front disclosure of any | ||
| illustrative figures. When ``can_call_tools`` is true it also forbids | ||
| supplying a fact-carrying tool-call argument that no source provides, | ||
| while leaving arguments the model is expected to compose untouched -- | ||
| except for a fact value written literally inside composed code or | ||
| document text, which the sourcing requirement still covers. | ||
| A prompt fragment forbidding unsupported claims and unsourced values of | ||
| every kind it enumerates, requiring the gap be reported instead, and | ||
| confining unsourced content to a current request that explicitly asks | ||
| for a template or sample. When ``can_call_tools`` is true it also | ||
| forbids supplying a fact-carrying tool-call argument that no source | ||
| provides, while leaving arguments the model is expected to compose | ||
| untouched -- except for a fact value written literally inside composed | ||
| code or document text, which the sourcing requirement still covers. | ||
| """ | ||
| insufficient_context_rule = ( | ||
| "If available context is insufficient, say so or use an appropriate " | ||
|
|
@@ -61,17 +67,15 @@ def grounding_rule(*, can_call_tools: bool = True) -> str: | |
| "messages, the retrieved context, or a value an earlier tool result " | ||
| "actually returned; never guess one, never substitute a " | ||
| "plausible-looking placeholder for one the user has not given, and " | ||
| "never carry one over from a different record. This does not restrict " | ||
| "values you are expected to compose yourself, such as a search query, " | ||
| "code or a command you write to do the work, a message or answer you " | ||
| "write to the user, or document text you were asked to produce. A fact " | ||
| "value written literally inside such composed code or text is still " | ||
| "subject to the sourcing rule above. The rule also does not reach a " | ||
| "default or inferred parameter value such as a page size or result " | ||
| "limit, which you are expected to decide yourself. Treat " | ||
| "a fact-carrying value you cannot source as " | ||
| "missing information rather than inventing it, and omit it when the " | ||
| "tool allows it to be omitted." | ||
| "never carry one over from a different record. The answer you write " | ||
| "to the user reaches you as an argument too; it is wording you " | ||
| "compose, so this argument standard does not reach it, while the " | ||
| "sourcing rule above still governs every fact inside it. This clause " | ||
| "does not reach a default or inferred parameter value such as a page " | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Minor] |
||
| "size or result limit, which you are expected to decide yourself. " | ||
| "Treat a fact-carrying value you cannot source as missing information " | ||
| "rather than inventing it, and omit it when the tool allows it to be " | ||
| "omitted." | ||
| if can_call_tools | ||
| else "" | ||
| ) | ||
|
|
@@ -81,12 +85,26 @@ def grounding_rule(*, can_call_tools: bool = True) -> str: | |
| "statistics, percentages, table rows, or time series) that are not " | ||
| "supported by the conversation, retrieved context, or tool results. " | ||
| f"{insufficient_context_rule}" | ||
| "Never invent figures to fill a gap, and never present invented numbers " | ||
| "as real data; produce unsupported figures only when the user explicitly " | ||
| "asked for a template, mockup, or illustrative example. Labeling is " | ||
| "required either way: if the answer ends up containing any figure that no " | ||
| "tool result or provided context supports, whether or not the user asked " | ||
| "for one, say so up front, before presenting it, and state that those " | ||
| "figures are illustrative placeholders not drawn from any data source." | ||
| f"Never fill a gap with an invented value, whether it is {VALUE_KINDS}: " | ||
| "when nothing in this conversation, the provided context, or a tool " | ||
| "result supports a value the answer needs, leave that value out and " | ||
| "say plainly that it is missing, rather than supplying one that " | ||
| "looks right. This does not restrict the wording you compose -- how " | ||
| "you phrase your reply, a search query, code or a command you write " | ||
| "to do the work, or document text you were asked to produce -- it " | ||
| "restricts every fact asserted inside that wording. A fact value " | ||
| "written literally inside such composed code or text is still " | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Minor] "the text you compose is yours; {VALUE_KINDS} that you place inside it is not" reads, on first parse, as the trailing relative clause ("that you place inside it") binding only to the last list item ("a row of a table"), not the full six-item disjunction. Recoverable from context but worth a comma/rephrase for readability. |
||
| "subject to the sourcing rule above: the text you compose is yours; " | ||
| f"{VALUE_KINDS} that you place inside it is not. The only case in " | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Minor] The template/sample exception is scoped to "in the answer" (stated twice, lines 98-105). When a user explicitly asks for template content that must go into a tool argument rather than the chat answer (e.g. "write a sample invoice and save it to a file"), the exception's literal wording doesn't reach that content, and no other clause broadens it. This is a newly-sharpened gap — the base wording wasn't scoped to "the answer". No test covers this scenario. Suggest extending the exception to cover composed tool-argument content too. |
||
| "which content that no source supports may appear in the answer is a " | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Minor] The template-exception trigger phrase "a template, a sample, or content that is not meant to be real" — "a sample" alone is ambiguous (could mean "an example populated with real data", not just mockup content); "explicitly asks" constrains that the user asked, not what they meant by "sample". No test covers this ambiguous-intent edge case. |
||
| "current user request that explicitly asks you to write a template, " | ||
| "a sample, or content that is not meant to be real; in that case, " | ||
| "before any of that content appears in the answer, state that the " | ||
| "request asked for content that is not real and that none of it " | ||
| "comes from a data source, and keep such content to what the request " | ||
| "asked for. Outside that case a caveat does not make an invented " | ||
| "value acceptable: if you find yourself about to add a note " | ||
| "explaining that some values are not real, remove those values and " | ||
| "report the gap instead." | ||
| f"{tool_argument_rule}" | ||
| ) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Major, Blocking — pointer] This hunk correctly broadens the compaction summary to preserve names/orgs/statuses/dates, not just numeric figures. But two other prompt strings in this same file were NOT touched and still only mention "figure": the non-LLM compaction summary trailer at execution.py:1245 and
_dropped_tool_results_noticeat execution.py:1412 (both unchanged since base commit f69c4ec, confirmed outside this PR's diff so not directly commentable here). Both tell the model to treat a missing/unsupported value as unavailable using only "figure"/"value, figure, statistic, table row, quotation, or identifier" — missing the name/org/date/status types this PR fixes everywhere else it touches (grounding.py, auto.py, dag.py, and this hunk). Trigger: any compaction event in a long-running session. Fix: broaden both sentences to the same value-kind scope used here.