Skip to content

fix(server): raise interactive compliance timeout to fit 67-storyboard suite #6803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bokelley merged 2 commits into from
Aug 23, 2026
Merged

fix(server): raise interactive compliance timeout to fit 67-storyboard suite #6803

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f89020b
fix(server): raise interactive compliance timeout to fit 67-storyboar…
garvitkaushik-123 Aug 23, 2026
07243af
test(server): verify interactive compliance timeout wiring
bokelley Aug 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions server/src/addie/mcp/member-tools.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ import { getTestKitForStoryboard } from '../../services/storyboards.js';
import {
hostedComplianceTarget,
hostedComplianceOptions,
HOSTED_FULL_COMPLIANCE_TIMEOUT_MS,
HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS,
hostedAuthProbeTaskForProfile,
withHostedStoryboardRunOptions,
withHostedTestOptions,
Expand Down Expand Up @@ -4650,7 +4650,7 @@ export function createMemberToolHandlers(

const complyOptions: ComplyOptions = {
test_session_id: `quality-eval-${Date.now()}`,
timeout_ms: HOSTED_FULL_COMPLIANCE_TIMEOUT_MS,
timeout_ms: HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS,
auth: authOption,
};
if (tracks) complyOptions.tracks = tracks;
Expand Down
8 changes: 8 additions & 0 deletions server/src/services/hosted-compliance-version.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,14 @@ export const HOSTED_COMPLIANCE_TARGET_PREFERENCE = [
// a single hung call hold a connection for 10 minutes.
export const HOSTED_FULL_COMPLIANCE_TIMEOUT_MS = 600_000;

// Interactive full-suite runs (evaluate_agent_quality via Addie) need a higher
// ceiling: 67 storyboards × 17s worst-case step pacing = ~1139s, plus headroom
// for target discovery and network jitter. The background callers (heartbeat,
// registry refresh) keep the shared 600s constant so heartbeat lock TTLs and

@aao-secretariat aao-secretariat Bot Aug 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium: The heartbeat is not just a lock-TTL concern — it is the canonical badge writer. compliance-heartbeat.ts:98 runs the same full storyboard suite at HOSTED_FULL_COMPLIANCE_TIMEOUT_MS (600s) and records canonical state; on timeout its catch path writes overall_status: 'failing' with replace_storyboard_statuses: true (compliance-heartbeat.ts:313-329), explicitly to purge stale passing data. If the 67-storyboard suite structurally needs ~1139s as this comment argues, the heartbeat cannot complete it at 600s either — so an evaluate_agent_quality run (now 1200s) can bring an agent to passing, and the next heartbeat cycle knocks it back to "Timed out / failing" and fires a badge-downgrade notification. The badge flaps every 12h. Confirm the heartbeat's per-step pacing actually clears 600s for the same suite; if it doesn't, the interactive raise needs to extend to the heartbeat (with the lock-TTL math at compliance-heartbeat.ts:75-76 recomputed) rather than diverging.

// hung-call risk stay bounded.
// TODO(adcontextprotocol/adcp-client#2221): collapse when upstream per-call timeout is restored
export const HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS = 1_200_000;

export interface HostedComplianceTarget {
requested: string;
version: string;
Expand Down
2 changes: 2 additions & 0 deletions server/tests/unit/storyboards.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import {
DEFAULT_HOSTED_COMPLIANCE_LINE,
DEFAULT_HOSTED_COMPLIANCE_VERSION,
HOSTED_FULL_COMPLIANCE_TIMEOUT_MS,
HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS,
badgeEligibleVersionsForHostedComplianceTarget,
hostedAuthProbeTaskForProfile,
hostedStaticApiKeyForProfile,
Expand Down Expand Up @@ -194,6 +195,7 @@ describe('wrapper contract', () => {
expect(DEFAULT_HOSTED_COMPLIANCE_VERSION).toBe('3.0.25');
expect(DEFAULT_HOSTED_COMPLIANCE_LINE).toBe('3.0');
expect(HOSTED_FULL_COMPLIANCE_TIMEOUT_MS).toBe(600_000);
expect(HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS).toBe(1_200_000);
expect(target.requested).toBe(DEFAULT_HOSTED_COMPLIANCE_LINE);
expect(target.version).toBe(DEFAULT_HOSTED_COMPLIANCE_VERSION);
expect(target.version).toMatch(/^3\.0\.\d+$/);
Expand Down
8 changes: 8 additions & 0 deletions tests/addie/member-tools.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ import { MEMBER_TOOLS, createMemberToolHandlers } from '../../server/src/addie/m
import { getGitHubAccessToken } from '../../server/src/services/pipes.js';
import { AgentContextDatabase } from '../../server/src/db/agent-context-db.js';
import { ComplianceDatabase } from '../../server/src/db/compliance-db.js';
import { HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS } from '../../server/src/services/hosted-compliance-version.js';
import { AgentSnapshotDatabase } from '../../server/src/db/agent-snapshot-db.js';
import * as wgService from '../../server/src/services/working-group-membership-service.js';

Expand Down Expand Up @@ -1164,6 +1165,13 @@ describe('createMemberToolHandlers', () => {

expect(result).toContain('Quality Evaluation: Seller Agent');
expect(result).not.toContain('diagnostic only');
expect(memberToolMocks.comply).toHaveBeenCalledWith(
expect.any(String),
expect.objectContaining({
timeout_ms: HOSTED_INTERACTIVE_COMPLIANCE_TIMEOUT_MS,
}),
expect.objectContaining({ requested: '3.0' }),
);
expect(ComplianceDatabase.prototype.recordComplianceRun).toHaveBeenCalledTimes(1);
expect(ComplianceDatabase.prototype.recordComplianceRun).toHaveBeenCalledWith(
expect.objectContaining({
Expand Down
Loading