Skip to content

test(interaction-harness): too little to judge is named starvation #10788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cixzhang merged 2 commits into from
Nov 9, 2026
Merged

test(interaction-harness): too little to judge is named starvation #10788

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e1f9922
test(interaction-harness): too little to judge is named starvation
vjeux Nov 9, 2026
1c1bef1
Merge main
vjeux Nov 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions internal/interaction-harness/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,11 +321,15 @@ decodes each frame once. A proof reads frames up to
decode only those. Pass either as `control`: the component passes when it moved on every frame the control
moved and its longest stall is no longer than the control's. Both are read from
the same screencast frames, so a compositor animation misses none. A window in
which the control moved on fewer than four frames fails, since the machine
presented too little to judge; the failure names the cast's frames in the
window and its longest gap. When no frame came for longer than
`FRAME_STARVATION_MS` there too, the machine starved the run, and
`busyWindowProof` throws `FrameStarvationError` instead of returning a verdict.
which the control moved on fewer than four frames is under the floor. Under the
floor, a run with no component failure on the frames shown is too little to
judge: `busyWindowProof` throws `FrameStarvationError` instead of returning a
verdict. The error names the control's moves, the cast's frames in the window
and its longest gap. A component that missed a frame the control moved on, or
stalled longer than it, failed on frames the compositor presented, so the
verdict keeps that red. A run the machine starved, with no frame for longer
than `FRAME_STARVATION_MS`, throws either way. `maxMedianFrameMs` judges the
cast's pace, not the component, so under the floor it keeps no red.
A run of frames with no new position counts as
moved when the change that ends it is a single cast step: the component moves
under one step a frame there, as on an easing tail, which the cast cannot show
Expand Down Expand Up @@ -409,8 +413,13 @@ belongs to its run. A run whose control moved on fewer frames than
and the pool leaves it out, so it never counts as a pass. A run whose block
began too late for its motion is left out the same way: its `runs` entry
is null, and `late` holds its `BusyWindowTooLateError`. The pool reads each
run without the starvation throw. The pool fails with
fewer than `minJudgedRuns` judged runs (half of all the runs, rounded up). In each judged run,
run without the starvation throw. With fewer than `minJudgedRuns` judged runs
(half of all the runs, rounded up), or a control that moved on fewer than four
frames over them, the pool is under the floor. Under the floor, a pool with no
component failure on the frames shown is too little to judge, and it throws
`FrameStarvationError`. A pool left short of judged runs by a run that began
too late fails instead, and so does a pool whose judged runs show a component
red. In each judged run,
the component must move on every frame its control moved on, so a freeze fails
its run.

Expand Down
177 changes: 155 additions & 22 deletions internal/interaction-harness/src/busy.control.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,14 +313,67 @@ describe('busyWindowProof against a control', () => {
);
});

it('fails a window in which the control itself presented too little to judge', () => {
it('names a window in which the control itself presented too little to judge as starvation, though every frame came', () => {
const still = asControl(
moving().map(f => ({...f, box: {...f.box!, y: 0}}) as TrackedFrame),
);
const proof = busyWindowProof(moving(), BLOCK, {control: still});
const run = () => busyWindowProof(moving(), BLOCK, {control: still});
expect(run).toThrow(FrameStarvationError);
// It names how starved the cast was: here every frame came, 60 Hz.
expect(run).toThrow(
/busyWindowProof: the page drew \d+ frames while a 700 ms block held the main thread, and the control moved on 0 of them, where the proof needs 4\. No frame came for 1[67]\.\d ms: the compositor presented too little in this window to judge/,
);
});

it("names a window under the control's floor as starvation when no frame came for less than 50 ms: a loaded machine's cast, its component on every frame the control moved on", () => {
// A 152 ms block: the cast delivered 8 frames, none for 44 ms. The control
// moved on 3 of them, as on a loaded runner.
const block = {from: 1000, to: 1152};
const times = [1008, 1028, 1048, 1092, 1104, 1116, 1128, 1146];
const moves = new Set([1048, 1092, 1128]);
let y = 0;
const loaded = [
...moving().filter(f => f.t < block.from),
...times.map(t => ({t, box: null}) as TrackedFrame),
...moving().filter(f => f.t > block.to),
].map(f => {
if (f.t < block.from || f.t > block.to || moves.has(f.t)) {
y += 3;
}
return {...f, box: {x: 0, y, width: 10, height: 10}} as TrackedFrame;
});
const run = () =>
busyWindowProof(loaded, block, {control: asControl(loaded)});
expect(run).toThrow(FrameStarvationError);
expect(run).toThrow(
/busyWindowProof: the page drew 8 frames while a 152 ms block held the main thread, and the control moved on 3 of them, where the proof needs 4\. No frame came for 44\.0 ms: the compositor presented too little in this window to judge/,
);
});

it('still fails a main-thread component under the floor: it misses the frames the control moved on, and the cast kept pace', () => {
const block = {from: 1000, to: 1090};
const frozen = moving().map(f =>
f.t > block.from && f.t < block.to + 20
? ({...f, box: {...f.box!, y: 3 * at(block.from)}} as TrackedFrame)
: f,
);
const proof = busyWindowProof(frozen, block, {
control: asControl(moving()),
});
expect(proof.verdict.failures.join('\n')).toMatch(
/the control moved on 0 frames in the judged part, under 4: the compositor presented too little in this window to judge \(the cast delivered \d+ frames in the \d+ ms block, none for 1[67] ms\)/,
/moved on 0 of the 3 frames the control moved/,
);
});

it("names a window under the floor as starvation when only the cast's median interval fails: the median is the cast's pace, not the component", () => {
const run = () =>
busyWindowProof(moving(), short(1000), {
control: asControl(moving()),
maxMedianFrameMs: 10,
});
expect(run).toThrow(FrameStarvationError);
expect(run).toThrow(
/the control moved on 3 of them, where the proof needs 4/,
);
});

Expand Down Expand Up @@ -448,15 +501,29 @@ function starvedCast(): TrackedFrame[] {
);
}

/** The numbers a `FrameStarvationError` from `run` names. */
function starvationNumbers(run: () => unknown) {
try {
run();
} catch (error) {
if (error instanceof FrameStarvationError) {
return error.numbers;
}
throw error;
}
throw new Error('no FrameStarvationError');
}

/** A 90 ms block from `from`: its judged part holds three 60 Hz frames. */
const short = (from: number) => ({from, to: from + 90});

describe('pooledBusyProof', () => {
it('judges a block too short for one run over pooled runs: one run cannot decide, three runs of a component that moves with the control pass', () => {
const control = asControl(moving());
const single = busyWindowProof(moving(), short(1000), {control});
expect(single.verdict.failures.join('\n')).toMatch(
/the control moved on 3 frames in the judged part, under 4/,
const single = () => busyWindowProof(moving(), short(1000), {control});
expect(single).toThrow(FrameStarvationError);
expect(single).toThrow(
/the control moved on 3 of them, where the proof needs 4/,
);
const runs = [1000, 1300, 1600].map(from => ({
tracked: moving(),
Expand Down Expand Up @@ -623,20 +690,22 @@ describe('pooledBusyProof', () => {
}
});

it('judges the control over the pool too: runs in which it presented too little fail', () => {
it('names a pool whose control presented too little in every run as starvation', () => {
const still = asControl(
moving().map(f => ({...f, box: {...f.box!, y: 0}}) as TrackedFrame),
);
const pooled = pooledBusyProof(
[1000, 1300].map(from => ({
tracked: moving(),
control: still,
window: short(from),
})),
const run = () =>
pooledBusyProof(
[1000, 1300].map(from => ({
tracked: moving(),
control: still,
window: short(from),
})),
);
expect(run).toThrow(FrameStarvationError);
expect(run).toThrow(
/pooledBusyProof: .*0 of 2 runs judged, under 1, and the control moved on 0 frames over them: the compositor presented too little in the others to judge/,
);
expect(pooled.verdict.failures).toEqual([
'0 of 2 runs judged, under 1: the compositor presented too little in the others to judge',
]);
});

/** `frames` held at the position they had at `from`, from `from` to `to`. */
Expand Down Expand Up @@ -665,20 +734,84 @@ describe('pooledBusyProof', () => {
);
});

it('fails a pool in which every run but one was too little to judge, even when that run holds enough frames', () => {
it('names a pool in which every run but one was too little to judge as starvation, even when that run holds enough frames', () => {
const windows = [1000, 1300, 1600].map(from => ({from, to: from + 200}));
const starved = held(held(moving(), 1300, 1500), 1600, 1800);
const pool = () =>
pooledBusyProof(
windows.map(window => ({
tracked: moving(),
control: asControl(starved),
window,
})),
);
expect(pool).toThrow(FrameStarvationError);
expect(pool).toThrow(
/pooledBusyProof: the page drew \d+ frames while 3 blocks held the main thread, and 1 of them had enough control frames to judge, where the proof needs 2\. .*1 of 3 runs judged, under 2, and the control moved on \d+ frames over them: the compositor presented too little in the others to judge/,
);
});

it('names a pool of four runs with one judged as starvation: its component moved on every frame its control moved on', () => {
const windows = [1000, 1250, 1500, 1750].map(short);
const control = asControl(
held(held(held(moving(), 1250, 1340), 1500, 1590), 1750, 1840),
);
const pool = () =>
pooledBusyProof(
windows.map(window => ({tracked: moving(), control, window})),
);
expect(pool).toThrow(FrameStarvationError);
expect(pool).toThrow(/1 of 4 runs judged, under 2/);
// Short of judged runs, the error counts every block's frames.
const numbers = starvationNumbers(pool);
expect(numbers.frames).toBe(20);
expect(numbers.longestGapMs).toBeCloseTo(FRAME, 1);
});

it('names the longest gap of every block in a pool short of judged runs, where the unjudged blocks drew no frame', () => {
const windows = [1000, 1250, 1500, 1750].map(short);
const control = asControl(
held(held(held(moving(), 1250, 1340), 1500, 1590), 1750, 1840),
);
const tracked = moving().filter(
f => !windows.slice(1).some(w => f.t > w.from && f.t < w.to),
);
const numbers = starvationNumbers(() =>
pooledBusyProof(windows.map(window => ({tracked, control, window}))),
);
expect(numbers.frames).toBe(5);
expect(numbers.longestGapMs).toBeCloseTo(90, 5);
});

it("names a pool short of judged runs as starvation when only the runs' median intervals fail", () => {
const windows = [1000, 1250, 1500, 1750].map(short);
const control = asControl(
held(held(held(moving(), 1250, 1340), 1500, 1590), 1750, 1840),
);
const pool = () =>
pooledBusyProof(
windows.map(window => ({tracked: moving(), control, window})),
{maxMedianFrameMs: 10},
);
expect(pool).toThrow(FrameStarvationError);
expect(pool).toThrow(/1 of 4 runs judged, under 2/);
});

it('still fails a pool short of judged runs when its judged run shows a component red', () => {
const windows = [1000, 1250, 1500, 1750].map(short);
const control = asControl(
held(held(held(moving(), 1250, 1340), 1500, 1590), 1750, 1840),
);
const pooled = pooledBusyProof(
windows.map(window => ({
tracked: moving(),
control: asControl(starved),
tracked: held(moving(), 1000, 1090),
control,
window,
})),
);
expect(pooled.judged).toEqual([true, false, false]);
expect(pooled.control.moved).toBeGreaterThanOrEqual(4);
expect(pooled.judged).toEqual([true, false, false, false]);
expect(pooled.verdict.failures.join('\n'), pooled.summary).toMatch(
/1 of 3 runs judged, under 2: the compositor presented too little in the others to judge/,
/run 1: moved on 0 of the 3 frames its control moved/,
);
});

Expand Down
Loading
Loading