Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,18 +25,18 @@ node scripts/check-links.js 'https://example.com/' ['<url>'…] # each o

1. A skill drives the browser through Playwright MCP and writes `kosh-reports/data/qa-report-<type>.json`. Meanwhile `collectors/capture.js` records every page the browser loads (timings, LCP and CLS, console errors, failed requests, tagging `byModel` those the model's own `browser_evaluate` made, from Chrome's initiator stack, and an inventory of its headings, links, images and share tags, clipped and capped, read again in Node since the page answers in its own world) into `kosh-reports/data/capture-<session id>.jsonl`. `scripts/check-links.js` records each off-site link's result beside it, in `links-<session id>.jsonl`.
2. `scripts/run-qa-report.sh` then, for that skill's own output:
- merges the capture (`scripts/page-capture.js merge`), setting `timestamp` to when the run's browser first loaded a page (the model's own value only if the capture has none and it's a date), and for performance rebuilds `pages[]` from it, replacing anything the model wrote, and refuses the run if its browser wasn't held to the site while it measured (the off-host wall, below); for functional it builds `inventory[]` from the capture (not `pages[]`, which `looksLikePerformance` would read as a performance report), adds the link checker's results as `linkChecks[]`, numbers the findings `F-01` onwards, refuses a `fixAt` of kind `file` that names a file, or a line, the run's folder doesn't have, and, like performance, refuses the run if its browser wasn't held to the site;
- merges the capture (`scripts/page-capture.js merge`), setting `timestamp` to when the run's browser first loaded a page (the model's own value only if the capture has none and it's a date), and for performance rebuilds `pages[]` from it, replacing anything the model wrote, and refuses the run if its browser wasn't held to the site while it measured (the off-host wall, below); for functional it builds `inventory[]` from the capture (not `pages[]`, which `looksLikePerformance` would read as a performance report), adds the link checker's results as `linkChecks[]` and the site's published pages the run never reached as `published` (below), numbers the findings `F-01` onwards, refuses a `fixAt` of kind `file` that names a file, or a line, the run's folder doesn't have, and, like performance, refuses the run if its browser wasn't held to the site; the validator also refuses a functional report holding `linkChecks[]` or `published` with no `captureSha256`, so a run whose browser recorded nothing but checked links is refused, and one that skipped the merge can't pass off a hand-written list as kosh's;
- stamps `provenance` (`scripts/stamp-provenance.js`): plugin version, skill and `collectors/` hashes, the Playwright MCP args, the model and effort from the session's transcript, and a seal;
- renders (`scripts/generate-report.js`), which validates against the schema first (`scripts/validate-report.js`) and refuses a mismatch, naming each field;
- archives the JSON and the capture in `kosh-reports/data/archive/` under the HTML's name, then deletes the source so the next run writes a new file, and for functional the session's link-check record, which the report now holds.
3. Any other report is stamped without the session's facts, rendered and copied to the archive, but never deleted, and nothing is merged.

Things that catch contributors out:

- A rendered report can be corrected by copying its archive JSON back to `kosh-reports/data/qa-report-<type>.json`, editing the text and running `run-qa-report.sh` again. It counts as a correction when its seal holds and an archived JSON has the same seal; a refused report is sealed too but never archived, so a retry still merges. The merge restores `url`, `pages[]`, `inventory[]` and `linkChecks[]` from that copy, whatever the edit did to them, and the seal keeps the timestamp and `heldTo`. It leaves the session's capture, one held for a retry and the link record for their own runs. Functional's findings are still numbered, so an ID can change between copies, and their `fixAt` checked. If a browser that isn't in the run's capture has loaded the site since, it's refused, since that's how a new run reusing an old report looks: finish the later run first, or save the copy under another name, which skips the merge. So does editing the archive copy in place, and nothing restores its measurements. Earlier copies move aside under the run's timestamp, then `-2` and so on, and every copy gets the capture its hash covers, even under a new website name.
- A rendered report can be corrected by copying its archive JSON back to `kosh-reports/data/qa-report-<type>.json`, editing the text and running `run-qa-report.sh` again. It counts as a correction when its seal holds and an archived JSON has the same seal; a refused report is sealed too but never archived, so a retry still merges. The merge restores `url`, `pages[]`, `inventory[]`, `linkChecks[]` and `published` from that copy, whatever the edit did to them, and the seal keeps the timestamp and `heldTo`. It leaves the session's capture, one held for a retry and the link record for their own runs. Functional's findings are still numbered, so an ID can change between copies, and their `fixAt` checked. If a browser that isn't in the run's capture has loaded the site since, it's refused, since that's how a new run reusing an old report looks: finish the later run first, or save the copy under another name, which skips the merge. So does editing the archive copy in place, and nothing restores its measurements. Earlier copies move aside under the run's timestamp, then `-2` and so on, and every copy gets the capture its hash covers, even under a new website name.
- `hooks/guard-capture.js` refuses edits to `capture*.jsonl` files and the link checker's `links*.jsonl` record in `kosh-reports/data/`, and anything naming this session's capture or record, since the merge seals both into a report as kosh's own. For commands that can reach a `kosh-reports/` kosh has run in (one holding a capture: anywhere in the session's project, or a run folder or a path in its `kosh-reports/` named by absolute path), it also refuses `*.jsonl` globs and any `rm` or `mv` of `kosh-reports/`, `kosh-reports/data`, the archive, or a glob of their contents, so clearing them is done by hand. Plugin hooks run on every Bash and edit call in every project, so `hooks/guard-capture.sh` starts Node only when the call's `command` or `file_path`, the only fields the guard reads, names `capture`, `links`, `jsonl` or `kosh-reports`, a word every rule needs; a rule that matches without one would be skipped, and the tests run every case through it. Without `node` on PATH it refuses such a call, since a guard that can't run shouldn't pass it.
- The merge keeps only the visits from the last browser that loaded the report's host, since a performance run starts with `browser_close`. A skill that closes the browser mid-run drops what it measured before.
- `inventory` also prints `published`: the site's published pages the run hasn't loaded or seen a link to, read from Node (the REST API, then WordPress's own sitemap, not a sitemap plugin's), because the menus can't show a page nothing links to. It asks only a host the run's browser has loaded, since the model writes the command's arguments, and keeps only http(s) URLs on that host, refusing an answer that redirected off it, since anything else is text nobody vetted. A source that leaves none falls through to the next, then to `error`, so a list that couldn't be read never looks like a site with nothing left. It uses query routes (`?rest_route=`, `?sitemap=`), which work on plain permalinks and stay inside a site in a subdirectory. It follows a redirect before checking the host, as `check-links.js` does, so neither stops a site sending a request to a private address. It isn't in the report, so a run that ignores it leaves no trace.
- `inventory` also prints `published`: the site's published pages the run hasn't loaded or seen a link to, read from Node (the REST API, then WordPress's own sitemap, not a sitemap plugin's), because the menus can't show a page nothing links to. It asks only a host the run's browser has loaded, since the model writes the command's arguments, and keeps only http(s) URLs on that host, refusing an answer that redirected off it, since anything else is text nobody vetted. A source that leaves none falls through to the next, then to `error`, so a list that couldn't be read never looks like a site with nothing left. It uses query routes (`?rest_route=`, `?sitemap=`), which work on plain permalinks and stay inside a site in a subdirectory. It follows a redirect before checking the host, as `check-links.js` does, so neither stops a site sending a request to a private address. The merge reads it again after the run and puts it in the report as `published`, so a run that skipped what it listed still shows it; `tests/run-qa-report.test.js` preloads `tests/no-network.js` into the scripts it runs, so its merges never ask a real site. It isn't read for a run refused for its wall.
- `kosh-reports/` is the project's, not kosh's: `$CLAUDE_PROJECT_DIR/kosh-reports`, which Claude Code sets for the capture and the hooks. Bash commands don't get it, so the scripts use the nearest `kosh-reports/data` at or above the cwd, looking no higher than the git root, else `kosh-reports/` in the cwd. The name is kosh's own so a project's `reports/` never gets client data mixed in. If `kosh-reports/.gitignore` is missing, the capture adds one (`*`) when it creates `data/`, and so does every successful render, even when Playwright made the folder first, since reports hold client URLs and console text. Schemas, skills and `plugin.json` still resolve from kosh's own folder. `KOSH_REPORTS_DIR` moves where the scripts and the capture look (the tests use it), not where the skills write.

## Playwright MCP (`plugin.json`)
Expand Down
45 changes: 45 additions & 0 deletions schemas/qa-report-functional-schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,51 @@
},
"description": "kosh's link checker's result for each off-site link checked this run. Added by scripts/run-qa-report.sh from the link checker's record — do not write this field."
},
"published": {
"description": "The site's published pages that this run neither loaded nor saw a link to, read from the site's REST API or sitemap after the run, or why they couldn't be read. Added by scripts/run-qa-report.sh — do not write this field.",
"oneOf": [
{
"type": "object",
"required": [
"source",
"total",
"notReached"
],
"additionalProperties": false,
"properties": {
"source": {
"enum": [
"REST API",
"sitemap"
]
},
"total": {
"type": "integer",
"minimum": 0
},
"notReached": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
{
"type": "object",
"required": [
"error"
],
"additionalProperties": false,
"properties": {
"error": {
"type": "string",
"minLength": 1
}
}
}
]
},
"explorationPass": {
"type": "array",
"description": "Threads investigated beyond the checks, recorded whatever the outcome, so an observation outside this test's scope survives here instead of becoming a finding. Findings themselves live in `issues`.",
Expand Down
19 changes: 18 additions & 1 deletion scripts/generate-report.js
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,23 @@ const visitedPagesList = visitedPages.length
? `<ul>${visitedPages.map((p) => `<li>${renderLink(p)}</li>`).join('')}</ul>`
: '<p class="muted">Not recorded.</p>';

const published = report.published && typeof report.published === 'object' ? report.published : null;
const notReached = Array.isArray(published?.notReached) ? published.notReached.filter((url) => typeof url === 'string') : [];
const publishedSummary = () => {
if (published.error) return `kosh couldn't read the site's published pages: ${escHtml(published.error)}.`;
const capped = published.source === 'REST API' && published.total > 100;
const none = capped ? 'This run loaded each of the 100 kosh read, or saw a link to it.' : 'This run loaded each one or saw a link to it.';
return `The site has ${escHtml(published.total)} published pages (from its ${escHtml(published.source)}${capped ? ', of which kosh read the newest 100' : ''}). ${notReached.length ? 'This run neither loaded these nor saw a link to them:' : none}`;
};
const publishedBlock = published
? `
<div class="visited-pages">
<h3>Published pages not reached</h3>
<p>Read from the site by kosh after the run, not reported by the model. ${publishedSummary()}</p>
${notReached.length ? `<ul>${notReached.map((url) => `<li>${renderLink(url)}</li>`).join('')}</ul>` : ''}
</div>`
: '';

// Core Web Vitals thresholds: https://web.dev/articles/vitals
const BANDS = { lcp: [2500, 4000], cls: [0.1, 0.25] };
const BAND_LABELS = { good: 'Good', fair: 'Needs improvement', poor: 'Poor' };
Expand Down Expand Up @@ -987,7 +1004,7 @@ ${explorationSection}
<div class="visited-pages">
<h3>Visited pages</h3>
${visitedPagesList}
</div>
</div>${publishedBlock}
</section>

<footer class="report-foot">
Expand Down
20 changes: 15 additions & 5 deletions scripts/page-capture.js
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ const refuseBadFixes = (report) => {
}
};

const RUN_FIELDS = ['url', 'pages', 'inventory', 'linkChecks'];
const RUN_FIELDS = ['url', 'pages', 'inventory', 'linkChecks', 'published'];
const captureBeside = (file) => file.replace(/\.json$/, '.capture.jsonl');

// A refused report is sealed too but never archived, so only a finished run's copy has a match here.
Expand Down Expand Up @@ -237,7 +237,7 @@ const correction = (reportPath, report, copyPath, flags) => {
if (functional) refuseBadFixes(report);
};

const merge = (reportPath, flags) => {
const merge = async (reportPath, flags) => {
const report = JSON.parse(fs.readFileSync(reportPath, 'utf8'));
const copyPath = sealedCopy(report);
if (copyPath) return correction(reportPath, report, copyPath, flags);
Expand Down Expand Up @@ -267,8 +267,15 @@ const merge = (reportPath, flags) => {
const functional = types.includes('functional');
if (functional) {
// Without a capture there's nothing to check visitedPages against, and the warning above says why.
if (visitCount(entries)) report.inventory = buildInventory(entries);
else delete report.inventory;
if (visitCount(entries)) {
report.inventory = buildInventory(entries);
// A run that skipped what inventory listed would otherwise leave no trace. Not for a run that's refused for its wall.
if (report.provenance?.heldTo) report.published = await published(report.url, report.inventory);
else delete report.published;
} else {
delete report.inventory;
delete report.published;
}
report.linkChecks = readLinkChecks(recordFile(), started);
numberFindings(report.issues);
}
Expand Down Expand Up @@ -297,6 +304,8 @@ const merge = (reportPath, flags) => {

const mergedFunctional = (report, entries) => {
console.log(`Page inventory merged: ${report.inventory?.length ?? 0} page × viewport rows from ${visitCount(entries)} visits on ${host(report.url)}, and ${report.linkChecks.length} link check${report.linkChecks.length === 1 ? '' : 's'}`);
const { published: listed } = report;
if (listed) console.log(listed.error ? ` Published pages not read: ${listed.error}` : ` Published pages not reached: ${listed.notReached.length} of ${listed.total}`);
refuseBadFixes(report);
// Only a fresh run is refused, as for performance: without a capture there's no inventory, and an old archive re-renders.
if (report.inventory?.length && !report.provenance?.heldTo) {
Expand Down Expand Up @@ -410,7 +419,8 @@ module.exports = { archiveText, badFileFixes, buildInventory, buildPages, host,

if (require.main === module) {
const [command, ...args] = process.argv.slice(2);
if (command === 'merge' && args[0]) merge(args[0], args.slice(1));
// run-qa-report.sh stops on this exit code, which an unhandled rejection doesn't always set.
if (command === 'merge' && args[0]) merge(args[0], args.slice(1)).catch((error) => { console.error(error.stack); process.exitCode = 1; });
else if (command === 'summary' && host(args[0])) summary(args);
else if (command === 'inventory' && host(args[0]) && args.every((url) => host(url))) inventory(args);
else {
Expand Down
4 changes: 4 additions & 0 deletions scripts/validate-report.js
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,10 @@ function checkReport(report, reportPath, types) {
if (rows?.length && !report.provenance?.captureSha256) {
ok = false;
console.error(`✗ ${field}[] has rows the page capture didn't write. ${path.join(__dirname, 'run-qa-report.sh')} builds them from the browser's record, and only for ${path.join(path.dirname(captureFile()), `qa-report-${type}.json`)}: save the report there without ${field}[], and re-run.`);
} else if (type === 'functional' && !report.provenance?.captureSha256 && (report.published || report.linkChecks?.length)) {
ok = false;
const added = report.published ? 'published' : 'linkChecks[]';
console.error(`✗ ${added} came without a page capture. ${path.join(__dirname, 'run-qa-report.sh')} adds it only to a report it merged from this session's capture, at ${path.join(path.dirname(captureFile()), `qa-report-${type}.json`)}. If a "no page capture" warning came first, restart Claude Code and run the skill again; otherwise save the report there without ${added}, and re-run.`);
} else if (missing.length) {
ok = false;
console.error(`✗ visitedPages lists pages the browser has no record of loading:\n${missing.map((url) => ` ${url}`).join('\n')}\nLoad each in the browser and re-run, or take it out of visitedPages if this run didn't test it.`);
Expand Down
Loading
Loading