Add an opaque governance record reference for decision operations - #368
AgentGymLeader wants to merge 12 commits into
Conversation
Add gen_ai.agent.decision.id, gen_ai.agent.decision.outcome, and gen_ai.agent.governance.ref as opt-in attributes on invoke_agent spans. These let telemetry correlate a decision point (a control resolving an attempted action) with a producer-owned governance record (policy, review, approval, or execution-context) via a payload-free opaque reference, without standardizing the referenced record's format or defining anything about actions that were never attempted. Refs: open-telemetry#239 Assisted-by: Claude Sonnet 5
…butes Runnable scenario exercising gen_ai.agent.decision.id, decision.outcome, and governance.ref on invoke_agent spans: an allowed attempt (child execute_tool span present) and a denied attempt (terminal block outcome, no child execute span, decision still recorded). The governance gate is a deliberately trivial in-process allowlist; the referenced context is an opaque producer-defined string, out of scope for the convention. Refs: open-telemetry#239 Assisted-by: Claude Sonnet 5 Assisted-by: OpenAI Codex
Assisted-by: Claude Sonnet 5
There was a problem hiding this comment.
Pull request overview
This pull request adds a small “governance decision join-point” surface to invoke_agent spans under gen_ai.agent.*, intended to let producers correlate an agent action attempt with an out-of-band governance/policy record via opaque identifiers, and includes a runnable reference scenario plus regenerated docs/reports.
Changes:
- Add new registry attributes:
gen_ai.agent.decision.id,gen_ai.agent.decision.outcome, andgen_ai.agent.governance.ref, and reference them from theinvoke_agentspan attribute group. - Add a new runnable reference scenario
reference/scenarios/agent-governance/and refresh reference reports/README to include it as a supporting library. - Regenerate generated documentation pages reflecting the new registry entries and span attribute tables.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| reference/scenarios/agent-governance/uv.lock | Locks Python dependencies for the new agent-governance reference scenario. |
| reference/scenarios/agent-governance/scenario.py | New reference scenario demonstrating allowed vs blocked governance decisions on invoke_agent spans. |
| reference/scenarios/agent-governance/pyproject.toml | Declares the scenario package metadata and dependencies. |
| reference/scenarios/agent-governance/data.json | Captures the scenario’s emitted span attribute coverage for report generation. |
| reference/reports/invoke-agent-internal-span.md | Regenerated report table reflecting the new attributes’ presence on the internal invoke-agent spec surface. |
| reference/reports/invoke-agent-client-span.md | Regenerated report showing agent-governance as a supporting library and listing the new attributes. |
| reference/reports/inference-span.md | Regenerated report including agent-governance inference coverage from the scenario’s chat span. |
| reference/reports/execute-tool-span.md | Regenerated report including agent-governance execute-tool coverage from the scenario. |
| reference/README.md | Updates the scenario/report index to include agent-governance where applicable. |
| model/gen-ai/spans.yaml | Adds governance-decision attribute refs to the shared invoke_agent attribute group. |
| model/gen-ai/registry.yaml | Adds registry definitions for the three new governance decision attributes (including the outcome enum members). |
| docs/registry/attributes/gen-ai.md | Generated registry docs updated to include the new attribute definitions and enum listing. |
| docs/gen-ai/gen-ai-agent-spans.md | Generated agent-span docs updated to include the new attributes and requirement levels. |
| changelog.d/368.enhancement.md | Towncrier fragment announcing the new governance decision join-point attributes. |
…ment, changelog wording Add an internal invoke_agent case to the agent-governance scenario so both span kinds referencing the common attribute group have reference coverage; stop emitting response identifiers on the invoke_agent span (they belong to the chat span); state requirement levels precisely in the changelog fragment. Refs: open-telemetry#368 Assisted-by: Claude Sonnet 5
lmolkova
left a comment
lmolkova
left a comment
There was a problem hiding this comment.
Is there a real library or component that's possible to instrument?
Manual-only instrumentations are not in scope of this project
…il runtime Replace the hand-rolled allowlist gate with real-library instrumentation: the scenario now drives the OpenAI Agents SDK with a plain-function input guardrail, and the governance decision attributes are read from the SDK's own guardrail runtime objects. The allowed run executes a function tool (child execute_tool span); the tripwire run records a terminal block with no model call and no child execute span. Refs: open-telemetry#368 Assisted-by: Claude Sonnet 5
|
@lmolkova The OpenAI Agents SDK is one: its guardrail runtime is a library-owned decision point. Reworked the scenario to instrument it, with a plain-function input guardrail (
|
|
@lmolkova Read #262 — I'll drop the overlap from this PR: What stays is one thing: a payload-free reference from a decision operation to the producer-owned record that governed it (policy, approval, execution context). Not telemetry about the check that ran — a join to the record behind it. The guardrail span is the natural first attachment, so I'll rework the PR into that shape. On real instrumentations: openai-agents exposes guardrail results on the run result, and ADK's tool callbacks look like a second candidate. Out-of-process decisions (proxy, approval gateway) would carry the same ref on their own tier's spans. |
…ence Drop gen_ai.agent.decision.id and gen_ai.agent.decision.outcome, and stop attaching decision attributes to invoke_agent spans. What remains is gen_ai.agent.governance.ref, defined to be recorded on the span representing the decision operation itself; its first attachment is the guardrail evaluation span proposed in open-telemetry#262, so span wiring and scenario emission land once that span type exists. The openai-agents guardrail scenario stays as the real-instrumentation base. Refs: open-telemetry#368 Assisted-by: Claude Sonnet 5
AgentGymLeader
changed the title
Pull request dashboard statusWaiting on the author · refreshed 2026-09-17 08:00 UTC Move out of draft to request review. Status above doesn't look right?
|
|
Producer-side input, since this is the layer we work on: a control plane in front of agent actions, which is the component that would emit this ref. Two notes on the attribute definition. First, worth writing the opacity rule into the definition itself, the same way the Second, for the audit case this attribute exists for, the useful guarantee is that the ref identifies the record as it stood when it governed the decision. Policies get edited and approvals get superseded. If the ref resolves to the current version of a record, a later edit silently changes what an old trace appears to say. A definition note that the ref should be stable to the record state at decision time (a version id, a snapshot id, whatever the producer uses) keeps the join trustworthy for a reviewer. That needs no mandated format and no hash, just the stability expectation stated. The sequencing also reads right to us: draft until #262, with the ref on the decision operation's span rather than copied onto agent spans. |
Resolves the three conflicts, all of which were in generated files: docs/registry/attributes/gen-ai.md, reference/README.md and reference/reports/invoke-agent-internal-span.md. model/gen-ai/registry.yaml, the source of truth, merged cleanly. Rather than hand-merging generated output, the conflicted files were reset to the upstream version and regenerated with the pinned weaver v0.25.1 via make generate-registry generate-docs, plus make generate-reference-reports.
|
The opaque reference now has its first downstream consumer drafting requirements for what sits behind the pointer: OpenSecureAIAlliance/RFCs#13 proposes machine-verifiable approval-to-execution scope binding for the SAFE incident-evidence compact, and frames itself explicitly against this PR's design ("the pointer exists and its contents are undefined," which it treats as the right division of labor). Linking it here so the two discussions stay aware of each other rather than forking. It reads as validation of the decision this PR made: the transport stays payload-free while record semantics evolve where they belong, and nothing in that issue asks this PR to change. |
|
@meshailabs Both go into the definition text. Opaque, producer-scoped, equality only, don't parse — that's the line that keeps the boundary from eroding, and #291's The stability point I had underweighted: a ref that resolves to the current version makes an old trace say what it didn't say at the time. Stating the expectation without mandating a format is the right amount. Neither is written yet — the registry entry follows the attachment span, which is still open (#262 closed, #427 carries it now). Thanks for the RFC link; agreed it asks nothing of this PR. |
Add the registry attribute and shared group to the eleven existing spans. Rename the reference scenario, emit the attribute on both runs, and regenerate the documentation and reference outputs. Assisted-by: GPT-5 Assisted-by: Claude Opus 5 Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com>
This PR replaces the earlier draft's attachment plan and does not depend on #427.
SDKs already expose guardrail evaluations as spans. A draft PR against python-genai (open-telemetry/opentelemetry-python-genai#621) maps the OpenAI Agents SDK's guardrail span data onto the span shape proposed in #427. That shape carries the verdict and little else, so the outcome and details of the evaluation stay with the component that ran it. A trace consumer needs one opaque attribute to join the span to whatever that component keeps. Nothing else here is standardized.
gen_ai.external_ref(string, development): an opaque, producer-defined identifier of an out-of-band record associated with the operation. Consumers compare it for equality only. The record's format, lifecycle, retrieval, and validation are outside this specification. meshailabs asked on this thread for the equality-only rule to be stated in the definition; it is. Producers are expected to keep the referent stable while the reference is in use; how is up to them.Defined once in
model/gen-ai/registry.yamland listed as recommended, when such a record exists for the operation, on the eleven spans already onmain(inference, embeddings, retrieval, fetch_response, memory, create_agent, invoke_agent client and internal, execute_tool, invoke_workflow, plan). The reference scenario shows it ongen_ai.invoke_agent.internalfor both the allowed run and the tripwire run. The python-genai PR above does not emit this attribute; it only shows that evaluation spans exist in a real SDK. If #427 merges, its run-guardrail span gets the same line.The join pattern is generic and has the same shape as
gen_ai.guardrail.security.external_finding_idin #427. What is specific is where it is defined and listed: gen_ai spans. It carries no permission-check semantics; that topic went to open-telemetry/semantic-conventions#4022 after #461. An external RFC (OpenSecureAIAlliance/RFCs#13, cited here on 2026-08-13) describes the same split: the pointer is defined, the content is not.The reference scenario is hand-authored. A second scenario from a real control-plane or guardrail component would be welcome.
Refs #239.