Skip to content

fix(server): raise interactive compliance timeout to fit 67-storyboard suite - #6803

Merged
bokelley merged 2 commits into
adcontextprotocol:mainfrom
garvitkaushik-123:fix/interactive-compliance-timeout
Aug 23, 2026
Merged

fix(server): raise interactive compliance timeout to fit 67-storyboard suite#6803
bokelley merged 2 commits into
adcontextprotocol:mainfrom
garvitkaushik-123:fix/interactive-compliance-timeout

Conversation

@garvitkaushik-123

@garvitkaushik-123 garvitkaushik-123 commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

The interactive full-compliance runner (evaluate_agent_quality via Addie) shared a 600s wall-clock budget with background callers. At observed step pacing of 11–17s, 600s fits at most ~35 storyboards out of 67, so the budget structurally prevents a full interactive grade regardless of seller performance.

Seller-side latency is not the bottleneck (measured ALB p50 at 13–19ms during affected runs); the 11–17s/step is dominated by runner-side orchestration overhead.

Changes

  • server/src/services/hosted-compliance-version.ts — add HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS = 1_200_000 (20 minutes). Leave HOSTED_FULL_COMPLIANCE_TIMEOUT_MS at 600s.
  • server/src/addie/mcp/member-tools.ts — use the interactive constant for evaluate_agent_quality.
  • server/tests/unit/storyboards.test.ts — pin the interactive constant alongside the existing 600s value.
  • tests/addie/member-tools.test.ts — assert that the handler passes the interactive timeout to comply().

Background callers (compliance-heartbeat.ts, registry-api.ts) stay on the shared 600s constant to preserve heartbeat lock TTLs and bound hung-call risk pending the upstream per-call timeout fix.

Verification

  • storyboards.test.ts: 48 passed
  • compliance-heartbeat.test.ts: 5 passed
  • member-tools.test.ts: 68 passed
  • No changeset needed: server/Addie-only change, no protocol or wire behavior

Refs #6632

garvitkaushik-123 and others added 2 commits August 24, 2026 00:06
…d suite

The interactive runner (evaluate_agent_quality via Addie) shared a 600s
wall-clock budget with background callers (heartbeat, registry refresh).
At observed step pacing of 11-17s, 600s fits at most 35 storyboards
out of 67, so sellers can never complete a full interactive grade.

Split the constant: introduce HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS
at 1200s for the interactive path while background callers keep the
shared 600s constant to preserve heartbeat lock TTLs and bound
hung-call risk.

Refs adcontextprotocol#6632
@bokelley

Copy link
Copy Markdown
Contributor

I pushed a maintainer follow-up adding a direct handler assertion that evaluate_agent_quality passes HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS into comply(). This protects the wiring as well as the constant value.

Verified with npx vitest run tests/addie/member-tools.test.ts — 68 passed.

// Interactive full-suite runs (evaluate_agent_quality via Addie) need a higher
// ceiling: 67 storyboards × 17s worst-case step pacing = ~1139s, plus headroom
// for target discovery and network jitter. The background callers (heartbeat,
// registry refresh) keep the shared 600s constant so heartbeat lock TTLs and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium: The heartbeat is not just a lock-TTL concern — it is the canonical badge writer. compliance-heartbeat.ts:98 runs the same full storyboard suite at HOSTED_FULL_COMPLIANCE_TIMEOUT_MS (600s) and records canonical state; on timeout its catch path writes overall_status: 'failing' with replace_storyboard_statuses: true (compliance-heartbeat.ts:313-329), explicitly to purge stale passing data. If the 67-storyboard suite structurally needs ~1139s as this comment argues, the heartbeat cannot complete it at 600s either — so an evaluate_agent_quality run (now 1200s) can bring an agent to passing, and the next heartbeat cycle knocks it back to "Timed out / failing" and fires a badge-downgrade notification. The badge flaps every 12h. Confirm the heartbeat's per-step pacing actually clears 600s for the same suite; if it doesn't, the interactive raise needs to extend to the heartbeat (with the lock-TTL math at compliance-heartbeat.ts:75-76 recomputed) rather than diverging.

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — server/Addie-only timeout split with no protocol surface touched.

What I checked

  • Adds HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS = 1_200_000 (20 min) for evaluate_agent_quality to fit the 67-storyboard suite, splitting it from the shared 600s constant. Reasoned and well-tested.
  • No static/schemas/source/**, docs/reference/**, dist/**, or task-definition changes — correctly changeset-free (app/Addie/operational scope only).
  • No breaking wire change, no oneOf regression, no released-artifact mutation, no 3.1.x eligibility concern.
  • high_risk is true only via server/src/addie/mcp/member-tools.ts (modified); the reviewer found no medium-or-higher concern on that file, so the modification is presumed safe.
  • gated_paths is false; review_decision: REVIEW_REQUIRED does not gate a non-gated path. No no-auto-approve team match.

Medium findings (1 — non-blocking)

  • server/src/services/hosted-compliance-version.ts:31 — Heartbeat (canonical badge writer) stays at 600s while the interactive path moves to 20 min; if the same suite can't finish at 600s in the background the badge may flap. Worth a follow-up but does not block.

One medium finding, no critical/high, no escalation trigger (medium is operability, not data-loss/schema/infra; high-risk file is modified but has no medium finding attached to it). Falls through to approve (row 9).

Medium findings

  • server/src/services/hosted-compliance-version.ts:31 — Heartbeat stays at 600s but is the canonical badge writer — divergence risks a flapping badge

@bokelley
bokelley enabled auto-merge (squash) August 23, 2026 19:18
@bokelley
bokelley self-requested a review August 23, 2026 19:26
@bokelley
bokelley merged commit b570105 into adcontextprotocol:main Aug 23, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants