billing_gate_dispatch selects and runs against agents missing its required_tools/interaction_model preconditions — unguarded step 1 fails hard instead of skipping

[EvgenyAndroid]

Summary

billing_gate_dispatch declares required_tools: [get_adcp_capabilities, sync_accounts] and agent.interaction_model: media_buy_seller, but the runner selects and executes it against agents that implement neither — a signals-only seller with no sync_accounts tool and no account/billing surface at all. The storyboard's first step is unguarded and asserts account.supported_billing unconditionally, so it fails hard on step 1 for every such agent instead of skipping via missing_tool, the way required_tools gating works correctly elsewhere (e.g. this same runner cleanly skips governance_setup/creative_reception/preview via missing_tool for a media-buy seller that doesn't implement optional tools).

The storyboard's later phases (capability_gate, per_agent_gate_reject, per_agent_gate_recover) all have proper test-kit-declaration guards and skip cleanly when their preconditions aren't met. Only the first step, get_capabilities under phase capability_discovery, has no guard — it presumably relies on required_tools gating the whole storyboard before step 1 is ever reached, which isn't happening.

Evidence

Tested against a live, production signals-only agent (https://adcp.signal-stack.io/mcp) — supported_protocols: ["signals"], specialisms: ["signal-owned"], agent.interaction_model: owned_signals per its own capabilities response, confirmed publicly with no auth required.

Tool list, confirmed via unauthenticated tools/list:

get_adcp_capabilities, get_signals, activate_signal, get_operation_status,
get_similar_signals, query_signals_nl, get_concept, search_concepts

No sync_accounts.

Ran in isolation with the storyboard's own declared prerequisites.test_kit (billing-gate-runner.yaml), @adcp/sdk@13.0.0, --compliance-version 3.1.15:

Storyboards: billing_gate_dispatch
Status: failing
Steps: 0 passed, 1 failed, 0 advisory validation(s) failed, 2 skipped
Skipped: not_applicable=2

❌ billing_gate_dispatch/get_capabilities (get_adcp_capabilities)
   At: /account/supported_billing
   Expected: account.supported_billing
   Actual: null

Re-ran with nova-motors.yaml (a different, mismatched test kit) to rule out a kit-selection artifact — byte-identical result. Step 1's assertion reads a field on the agent's own get_adcp_capabilities response and isn't influenced by which test kit is loaded, so this isn't a kit-selection issue.

The production registry card for this agent shows billing_gate_dispatch as partial in the aggregated default run rather than failing as in my isolated run — both are wrong in the same direction (neither is untested/skipped), and the discrepancy between them suggests aggregation-context also affects how this specific case gets classified, on top of the underlying selection bug.

Impact

Any signals-only (or otherwise sync_accounts-less) seller that is honest about not implementing account/billing has no way to get a clean skip on this storyboard — it hard-fails every run, contributing to a permanently degraded card for a capability gap the seller correctly never claimed. Same shape as the #6320 family of findings from earlier this month (a step the runner declines to run for a stated reason still counts against the seller), now confirmed on a second, independent production deployment and a different required_tools/interaction_model combination than those earlier reports covered.

Suggested

• Enforce required_tools as a storyboard-level selection gate before step 1 runs — if any declared required tool is absent from the agent's tool list, skip the whole storyboard via missing_tool, consistent with how this already works for optional per-step tools elsewhere.
• Alternatively/additionally, gate agent.interaction_model at selection time — a storyboard declared media_buy_seller shouldn't be selected against an agent whose own interaction_model (or specialisms) is signals-only.
• If neither selection-time gate is feasible short-term, add an explicit guard to billing_gate_dispatch/get_capabilities itself mirroring the later phases' pattern (skip when account.supported_billing is absent from the response, rather than asserting its presence).

Related

Filed while diagnosing a public registry card's billing_gate_dispatch: partial entry for adcp.signal-stack.io/mcp. A second storyboard flagged in the same card, error_compliance_signals, was investigated in parallel and found to be working as intended (7/7 steps pass; its silent/non-badge-passing status is the same "no stateful lifecycle observed" classification already accepted as by-design elsewhere) — not filed.

[bokelley]

Thanks for filing!

Triage

Classification: Bug — storyboard YAML authoring defect + SDK hardening opportunity
Bucket(s): conformance
Status: ready-to-implement

What the experts said:

• ad-tech-protocol-expert: required_tools deliberately uses OR ("at least one must be present") per types.ts:60; listing get_adcp_capabilities alongside sync_accounts makes the gate a no-op since every conformant agent advertises get_adcp_capabilities. Primary fix is a one-line YAML change in adcontextprotocol/adcp, not this repo. interaction_model is metadata only — the runner has no selection gate for it anywhere.
• code-reviewer: Confirms both gate sites (comply.ts:886, runner.ts:2611) correctly implement the OR contract. Flags two SDK-side hardening items that belong here: (1) a loader-time warning when required_tools lists a known universal tool, and (2) unit tests for both gate sites (currently zero coverage). Also flags the two sites share no helper and risk diverging.

My take: Primary fix is a YAML authoring change in adcontextprotocol/adcp — change billing_gate_dispatch's required_tools: [get_adcp_capabilities, sync_accounts] to required_tools: [sync_accounts] (drop the universal tool that trivially satisfies the OR gate). Reference #2623 and #2250 as prior art. Three non-breaking SDK hardening items are ready here independently.

Implementation brief (this repo — non-breaking, additive):

• src/lib/testing/storyboard/loader.ts (or YAML validator): emit a warning at load time when any required_tools entry is a known universally-present tool (get_adcp_capabilities), so storyboard authors catch the mistake before CI.
• Extract a shared hasRequiredTools(storyboard, agentTools): boolean helper used by both comply.ts:886 and runner.ts:2611 to prevent the two sites diverging silently.
• Unit tests: mock agent with ['get_adcp_capabilities'] only + storyboard with required_tools: ['sync_accounts'] → assert skip_reason: 'missing_tool', overall_passed: true at both gate sites.
• Run npm run ci:quick to validate; no public API changes.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01HRNFy81QnuvpeVrh7zNLVx

---

Generated by Claude Code