From 8f7aa983902538fbc4077f257f94dfb77a2769f8 Mon Sep 17 00:00:00 2001 From: katodea <17304002+katodea@users.noreply.github.com> Date: Fri, 6 Nov 2026 08:32:20 -0500 Subject: [PATCH] feat: list published pages a run never reached in the functional report --- AGENTS.md | 6 +-- schemas/qa-report-functional-schema.json | 45 +++++++++++++++++ scripts/generate-report.js | 19 ++++++- scripts/page-capture.js | 20 ++++++-- scripts/validate-report.js | 4 ++ tests/generate-report.test.js | 23 +++++++++ tests/no-network.js | 2 + tests/page-capture.test.js | 7 +++ tests/run-qa-report.test.js | 63 ++++++++++++++++++++++-- tests/validate-report.test.js | 13 +++++ 10 files changed, 190 insertions(+), 12 deletions(-) create mode 100644 tests/no-network.js diff --git a/AGENTS.md b/AGENTS.md index 05eb7d5..9fd0624 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,7 @@ node scripts/check-links.js 'https://example.com/' [''…] # each o 1. A skill drives the browser through Playwright MCP and writes `kosh-reports/data/qa-report-.json`. Meanwhile `collectors/capture.js` records every page the browser loads (timings, LCP and CLS, console errors, failed requests, tagging `byModel` those the model's own `browser_evaluate` made, from Chrome's initiator stack, and an inventory of its headings, links, images and share tags, clipped and capped, read again in Node since the page answers in its own world) into `kosh-reports/data/capture-.jsonl`. `scripts/check-links.js` records each off-site link's result beside it, in `links-.jsonl`. 2. `scripts/run-qa-report.sh` then, for that skill's own output: - - merges the capture (`scripts/page-capture.js merge`), setting `timestamp` to when the run's browser first loaded a page (the model's own value only if the capture has none and it's a date), and for performance rebuilds `pages[]` from it, replacing anything the model wrote, and refuses the run if its browser wasn't held to the site while it measured (the off-host wall, below); for functional it builds `inventory[]` from the capture (not `pages[]`, which `looksLikePerformance` would read as a performance report), adds the link checker's results as `linkChecks[]`, numbers the findings `F-01` onwards, refuses a `fixAt` of kind `file` that names a file, or a line, the run's folder doesn't have, and, like performance, refuses the run if its browser wasn't held to the site; + - merges the capture (`scripts/page-capture.js merge`), setting `timestamp` to when the run's browser first loaded a page (the model's own value only if the capture has none and it's a date), and for performance rebuilds `pages[]` from it, replacing anything the model wrote, and refuses the run if its browser wasn't held to the site while it measured (the off-host wall, below); for functional it builds `inventory[]` from the capture (not `pages[]`, which `looksLikePerformance` would read as a performance report), adds the link checker's results as `linkChecks[]` and the site's published pages the run never reached as `published` (below), numbers the findings `F-01` onwards, refuses a `fixAt` of kind `file` that names a file, or a line, the run's folder doesn't have, and, like performance, refuses the run if its browser wasn't held to the site; the validator also refuses a functional report holding `linkChecks[]` or `published` with no `captureSha256`, so a run whose browser recorded nothing but checked links is refused, and one that skipped the merge can't pass off a hand-written list as kosh's; - stamps `provenance` (`scripts/stamp-provenance.js`): plugin version, skill and `collectors/` hashes, the Playwright MCP args, the model and effort from the session's transcript, and a seal; - renders (`scripts/generate-report.js`), which validates against the schema first (`scripts/validate-report.js`) and refuses a mismatch, naming each field; - archives the JSON and the capture in `kosh-reports/data/archive/` under the HTML's name, then deletes the source so the next run writes a new file, and for functional the session's link-check record, which the report now holds. @@ -33,10 +33,10 @@ node scripts/check-links.js 'https://example.com/' [''…] # each o Things that catch contributors out: -- A rendered report can be corrected by copying its archive JSON back to `kosh-reports/data/qa-report-.json`, editing the text and running `run-qa-report.sh` again. It counts as a correction when its seal holds and an archived JSON has the same seal; a refused report is sealed too but never archived, so a retry still merges. The merge restores `url`, `pages[]`, `inventory[]` and `linkChecks[]` from that copy, whatever the edit did to them, and the seal keeps the timestamp and `heldTo`. It leaves the session's capture, one held for a retry and the link record for their own runs. Functional's findings are still numbered, so an ID can change between copies, and their `fixAt` checked. If a browser that isn't in the run's capture has loaded the site since, it's refused, since that's how a new run reusing an old report looks: finish the later run first, or save the copy under another name, which skips the merge. So does editing the archive copy in place, and nothing restores its measurements. Earlier copies move aside under the run's timestamp, then `-2` and so on, and every copy gets the capture its hash covers, even under a new website name. +- A rendered report can be corrected by copying its archive JSON back to `kosh-reports/data/qa-report-.json`, editing the text and running `run-qa-report.sh` again. It counts as a correction when its seal holds and an archived JSON has the same seal; a refused report is sealed too but never archived, so a retry still merges. The merge restores `url`, `pages[]`, `inventory[]`, `linkChecks[]` and `published` from that copy, whatever the edit did to them, and the seal keeps the timestamp and `heldTo`. It leaves the session's capture, one held for a retry and the link record for their own runs. Functional's findings are still numbered, so an ID can change between copies, and their `fixAt` checked. If a browser that isn't in the run's capture has loaded the site since, it's refused, since that's how a new run reusing an old report looks: finish the later run first, or save the copy under another name, which skips the merge. So does editing the archive copy in place, and nothing restores its measurements. Earlier copies move aside under the run's timestamp, then `-2` and so on, and every copy gets the capture its hash covers, even under a new website name. - `hooks/guard-capture.js` refuses edits to `capture*.jsonl` files and the link checker's `links*.jsonl` record in `kosh-reports/data/`, and anything naming this session's capture or record, since the merge seals both into a report as kosh's own. For commands that can reach a `kosh-reports/` kosh has run in (one holding a capture: anywhere in the session's project, or a run folder or a path in its `kosh-reports/` named by absolute path), it also refuses `*.jsonl` globs and any `rm` or `mv` of `kosh-reports/`, `kosh-reports/data`, the archive, or a glob of their contents, so clearing them is done by hand. - The merge keeps only the visits from the last browser that loaded the report's host, since a performance run starts with `browser_close`. A skill that closes the browser mid-run drops what it measured before. -- `inventory` also prints `published`: the site's published pages the run hasn't loaded or seen a link to, read from Node (the REST API, then WordPress's own sitemap, not a sitemap plugin's), because the menus can't show a page nothing links to. It asks only a host the run's browser has loaded, since the model writes the command's arguments, and keeps only http(s) URLs on that host, refusing an answer that redirected off it, since anything else is text nobody vetted. A source that leaves none falls through to the next, then to `error`, so a list that couldn't be read never looks like a site with nothing left. It uses query routes (`?rest_route=`, `?sitemap=`), which work on plain permalinks and stay inside a site in a subdirectory. It follows a redirect before checking the host, as `check-links.js` does, so neither stops a site sending a request to a private address. It isn't in the report, so a run that ignores it leaves no trace. +- `inventory` also prints `published`: the site's published pages the run hasn't loaded or seen a link to, read from Node (the REST API, then WordPress's own sitemap, not a sitemap plugin's), because the menus can't show a page nothing links to. It asks only a host the run's browser has loaded, since the model writes the command's arguments, and keeps only http(s) URLs on that host, refusing an answer that redirected off it, since anything else is text nobody vetted. A source that leaves none falls through to the next, then to `error`, so a list that couldn't be read never looks like a site with nothing left. It uses query routes (`?rest_route=`, `?sitemap=`), which work on plain permalinks and stay inside a site in a subdirectory. It follows a redirect before checking the host, as `check-links.js` does, so neither stops a site sending a request to a private address. The merge reads it again after the run and puts it in the report as `published`, so a run that skipped what it listed still shows it; `tests/run-qa-report.test.js` preloads `tests/no-network.js` into the scripts it runs, so its merges never ask a real site. It isn't read for a run refused for its wall. - `kosh-reports/` is the project's, not kosh's: `$CLAUDE_PROJECT_DIR/kosh-reports`, which Claude Code sets for the capture and the hooks. Bash commands don't get it, so the scripts use the nearest `kosh-reports/data` at or above the cwd, looking no higher than the git root, else `kosh-reports/` in the cwd. The name is kosh's own so a project's `reports/` never gets client data mixed in. If `kosh-reports/.gitignore` is missing, the capture adds one (`*`) when it creates `data/`, and so does every successful render, even when Playwright made the folder first, since reports hold client URLs and console text. Schemas, skills and `plugin.json` still resolve from kosh's own folder. `KOSH_REPORTS_DIR` moves where the scripts and the capture look (the tests use it), not where the skills write. ## Playwright MCP (`plugin.json`) diff --git a/schemas/qa-report-functional-schema.json b/schemas/qa-report-functional-schema.json index 48907e7..302f02b 100644 --- a/schemas/qa-report-functional-schema.json +++ b/schemas/qa-report-functional-schema.json @@ -125,6 +125,51 @@ }, "description": "kosh's link checker's result for each off-site link checked this run. Added by scripts/run-qa-report.sh from the link checker's record — do not write this field." }, + "published": { + "description": "The site's published pages that this run neither loaded nor saw a link to, read from the site's REST API or sitemap after the run, or why they couldn't be read. Added by scripts/run-qa-report.sh — do not write this field.", + "oneOf": [ + { + "type": "object", + "required": [ + "source", + "total", + "notReached" + ], + "additionalProperties": false, + "properties": { + "source": { + "enum": [ + "REST API", + "sitemap" + ] + }, + "total": { + "type": "integer", + "minimum": 0 + }, + "notReached": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + { + "type": "object", + "required": [ + "error" + ], + "additionalProperties": false, + "properties": { + "error": { + "type": "string", + "minLength": 1 + } + } + } + ] + }, "explorationPass": { "type": "array", "description": "Threads investigated beyond the checks, recorded whatever the outcome, so an observation outside this test's scope survives here instead of becoming a finding. Findings themselves live in `issues`.", diff --git a/scripts/generate-report.js b/scripts/generate-report.js index 8f7fdf6..1034dc3 100644 --- a/scripts/generate-report.js +++ b/scripts/generate-report.js @@ -349,6 +349,23 @@ const visitedPagesList = visitedPages.length ? `
    ${visitedPages.map((p) => `
  • ${renderLink(p)}
  • `).join('')}
` : '

Not recorded.

'; +const published = report.published && typeof report.published === 'object' ? report.published : null; +const notReached = Array.isArray(published?.notReached) ? published.notReached.filter((url) => typeof url === 'string') : []; +const publishedSummary = () => { + if (published.error) return `kosh couldn't read the site's published pages: ${escHtml(published.error)}.`; + const capped = published.source === 'REST API' && published.total > 100; + const none = capped ? 'This run loaded each of the 100 kosh read, or saw a link to it.' : 'This run loaded each one or saw a link to it.'; + return `The site has ${escHtml(published.total)} published pages (from its ${escHtml(published.source)}${capped ? ', of which kosh read the newest 100' : ''}). ${notReached.length ? 'This run neither loaded these nor saw a link to them:' : none}`; +}; +const publishedBlock = published + ? ` +
+

Published pages not reached

+

Read from the site by kosh after the run, not reported by the model. ${publishedSummary()}

+ ${notReached.length ? `
    ${notReached.map((url) => `
  • ${renderLink(url)}
  • `).join('')}
` : ''} +
` + : ''; + // Core Web Vitals thresholds: https://web.dev/articles/vitals const BANDS = { lcp: [2500, 4000], cls: [0.1, 0.25] }; const BAND_LABELS = { good: 'Good', fair: 'Needs improvement', poor: 'Poor' }; @@ -987,7 +1004,7 @@ ${explorationSection}

Visited pages

${visitedPagesList} -
+ ${publishedBlock}