fix(server): raise interactive compliance timeout to fit 67-storyboard suite - #6803
Conversation
…d suite The interactive runner (evaluate_agent_quality via Addie) shared a 600s wall-clock budget with background callers (heartbeat, registry refresh). At observed step pacing of 11-17s, 600s fits at most 35 storyboards out of 67, so sellers can never complete a full interactive grade. Split the constant: introduce HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS at 1200s for the interactive path while background callers keep the shared 600s constant to preserve heartbeat lock TTLs and bound hung-call risk. Refs adcontextprotocol#6632
|
I pushed a maintainer follow-up adding a direct handler assertion that Verified with |
| // Interactive full-suite runs (evaluate_agent_quality via Addie) need a higher | ||
| // ceiling: 67 storyboards × 17s worst-case step pacing = ~1139s, plus headroom | ||
| // for target discovery and network jitter. The background callers (heartbeat, | ||
| // registry refresh) keep the shared 600s constant so heartbeat lock TTLs and |
There was a problem hiding this comment.
Medium: The heartbeat is not just a lock-TTL concern — it is the canonical badge writer. compliance-heartbeat.ts:98 runs the same full storyboard suite at HOSTED_FULL_COMPLIANCE_TIMEOUT_MS (600s) and records canonical state; on timeout its catch path writes overall_status: 'failing' with replace_storyboard_statuses: true (compliance-heartbeat.ts:313-329), explicitly to purge stale passing data. If the 67-storyboard suite structurally needs ~1139s as this comment argues, the heartbeat cannot complete it at 600s either — so an evaluate_agent_quality run (now 1200s) can bring an agent to passing, and the next heartbeat cycle knocks it back to "Timed out / failing" and fires a badge-downgrade notification. The badge flaps every 12h. Confirm the heartbeat's per-step pacing actually clears 600s for the same suite; if it doesn't, the interactive raise needs to extend to the heartbeat (with the lock-TTL math at compliance-heartbeat.ts:75-76 recomputed) rather than diverging.
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — server/Addie-only timeout split with no protocol surface touched.
What I checked
- Adds
HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS = 1_200_000(20 min) forevaluate_agent_qualityto fit the 67-storyboard suite, splitting it from the shared 600s constant. Reasoned and well-tested. - No
static/schemas/source/**,docs/reference/**,dist/**, or task-definition changes — correctly changeset-free (app/Addie/operational scope only). - No breaking wire change, no oneOf regression, no released-artifact mutation, no 3.1.x eligibility concern.
high_riskis true only viaserver/src/addie/mcp/member-tools.ts (modified); the reviewer found no medium-or-higher concern on that file, so the modification is presumed safe.gated_pathsis false;review_decision: REVIEW_REQUIREDdoes not gate a non-gated path. No no-auto-approve team match.
Medium findings (1 — non-blocking)
- server/src/services/hosted-compliance-version.ts:31 — Heartbeat (canonical badge writer) stays at 600s while the interactive path moves to 20 min; if the same suite can't finish at 600s in the background the badge may flap. Worth a follow-up but does not block.
One medium finding, no critical/high, no escalation trigger (medium is operability, not data-loss/schema/infra; high-risk file is modified but has no medium finding attached to it). Falls through to approve (row 9).
Medium findings
- server/src/services/hosted-compliance-version.ts:31 — Heartbeat stays at 600s but is the canonical badge writer — divergence risks a flapping badge
Summary
The interactive full-compliance runner (
evaluate_agent_qualityvia Addie) shared a 600s wall-clock budget with background callers. At observed step pacing of 11–17s, 600s fits at most ~35 storyboards out of 67, so the budget structurally prevents a full interactive grade regardless of seller performance.Seller-side latency is not the bottleneck (measured ALB p50 at 13–19ms during affected runs); the 11–17s/step is dominated by runner-side orchestration overhead.
Changes
server/src/services/hosted-compliance-version.ts— addHOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS = 1_200_000(20 minutes). LeaveHOSTED_FULL_COMPLIANCE_TIMEOUT_MSat 600s.server/src/addie/mcp/member-tools.ts— use the interactive constant forevaluate_agent_quality.server/tests/unit/storyboards.test.ts— pin the interactive constant alongside the existing 600s value.tests/addie/member-tools.test.ts— assert that the handler passes the interactive timeout tocomply().Background callers (
compliance-heartbeat.ts,registry-api.ts) stay on the shared 600s constant to preserve heartbeat lock TTLs and bound hung-call risk pending the upstream per-call timeout fix.Verification
storyboards.test.ts: 48 passedcompliance-heartbeat.test.ts: 5 passedmember-tools.test.ts: 68 passedRefs #6632