Skip to content

design-qa.md

Latest commit

 

History

History
49 lines (30 loc) · 18.3 KB

design-qa.md

File metadata and controls

49 lines (30 loc) · 18.3 KB
Raw

Design QA — Agent-first annotation workbench

Reference and evidence

The desktop screenshots are full-viewport captures. A focused crop was unnecessary because each viewport shows the page canvas, Agent progress, and the pending decision together. At 390px, the title/type row and page controls occupy separate rows; the review candidate and page controls remain visible without horizontal overflow.

Design changes and comparison

The concept makes Agent progress and human review the workbench's first-order elements. The implementation keeps all four Agent modes visible above execution settings, along with current work, next action, page coverage, review count, and candidate decisions. Assist applies clear low- and medium-priority results and holds high-priority or uncertain results for a person. Autopilot applies every clear result, reports high-priority items, and waits only on uncertain evidence. Each candidate exposes add, edit, and reject actions; the page link and canvas now align to the active review candidate, including after restoring a saved pending workspace.

The export control now opens an explicit click menu with native output first, rather than resembling a hover-only menu while immediately downloading JSON. Document badges and units distinguish PDF pages, Word pages, PowerPoint slides, and Excel sheets. Key labels and controls are larger, advanced configuration is collapsed, and the in-app help includes the generated four-format workflow visual.

Excel now uses a readable worksheet grid with a sheet picker, a visible 16-column window, navigation across the first 80 preview columns, and a stated row/column preview limit. Change cards jump to visible target cells. Pending, approved, and rejected changes can load original values beside proposed values, including targets outside the grid. Large ranges page through bounded 5 × 8 slices; a small 5 × 8 target is kept wholly inside its context window. Context columns use Excel letters when the actual header row is unknown.

Findings and verification

Earlier review identified buried review controls, an ambiguous export menu, unclear document format, small labels, a mismatch between the pending candidate page and the canvas, and undisclosed Excel preview limits. Those issues are resolved in the current screenshots and UI. Final UI review found no remaining actionable P0–P2 issues at 390, 1280, or 1440px.

The production-preview browser E2E verifies PDF approve, correction, rejection, and JSON export, then reloads persisted run history, exports its event and human-decision record, and confirms a same-named PDF with a different source hash cannot see the old history or annotations. It keeps the cooling-fan PDF as the default and also opens the optional fictional termination-contract demo; the browser confirms the PDF contains no preset answer labels, the persistent notice identifies the scripted content as not AI-run and not legal advice, and reviewing the candidate does not hide that notice or call an AI endpoint. The separate live contract entry opens an unlabeled 11-page, 14-clause source, selects Autopilot, and requires a model connection before analysis. It checks all four mode controls at three viewport sizes, then runs Autopilot across the full document and confirms a clear high-priority finding is applied and reported without entering the review queue. It imports a browser folder with PDF, XLSX, image, and unsupported text files, checks the supported subset, batches only the selected PDF and workbook, and leaves the image unprocessed. Both the PDF ambiguity and an uncertain XLSX cell change remain in review, appear in the folder result count, and the workbook proposal restores when the XLSX is opened again. It then selects the image alone, confirms it renders, queues and resolves a mocked visual review candidate, and exports the folder-relative annotation record. DOCX and PPTX also go through the browser upload, mocked annotation, and native-export controls; a loopback API test independently converts minimal Office packages and inspects the returned Word comments and PowerPoint tag/shape ZIP parts. Item-only remains the correction default. During a saved full-document continuation it drafts a correction rule from candidate evidence, lets the reviewer edit and accept it, and confirms the versioned rule and its first page reach both remaining-page Agent requests and the Validator context. A delayed rule draft is discarded after the reviewer changes the corrected label. It reloads a saved approved annotation, runs a mocked read-only Validator review, checks the zero-finding state, confirms no annotation records or review outcomes changed and no annotate/run endpoint was called, then delays a Validator response while switching documents to verify that stale results are discarded. Annotator specialist tests cover visual-only evidence omitted from extracted text, explicit item-only decision scoping, bounded output, and the parent Human Review approval path; the specialist is read-only and limited to one delegation per page and 24 per run. Codex App Server workbook integration uses bounded host-mediated range reads and stages structured changes through the XLSX adapter. Suggest keeps every change pending; Assist holds high-priority changes for review, while Autopilot can apply clear high-priority changes and report them. The workbook UI exposes direct approve/reject for pending proposals, and the route rejects stale or concurrent decisions. Its loopback API test verifies the strict read/propose turns, a new column with values, clear auto-application versus Suggest review, Autopilot's clear high-priority auto-application, persistence across API restart, and restoration of saved proposals into a newly uploaded session for the same workbook. It rejects both a mismatched workbook hash and a stale per-record hash, then confirms restored review context, approval/rejection, and unchanged uploaded source bytes. A separate Agents SDK workbook API test runs column creation, two same-Run approvals, readback, native XLSX export, and independent ExcelJS reload; it checks exported values and source-byte immutability. An 80-page scripted adapter run verifies search/navigation to page 80 and same-Run Human Review resume. PPTX exporter tests verify that both Transitional and Strict OOXML use matching slide, shape, relationship, and tag namespaces while preserving the OPC relationship/content-type namespaces. It also uploads a generated wide XLSX fixture, registers proposed changes with the local API, and tests column navigation, a complete 40-cell context, multi-page context for a 100-cell range, and context on an approved change. Native workspace tests traverse nested temporary folders, skip ignored directories and symlinks, and read the selected source bytes through the same injected filesystem seam used by Tauri; the native picker now grants recursive access for descendant reads. AI responses are local fixtures; separate API integration tests use loopback fixtures to verify strict schema handling, memory-only evidence, Codex workbook approvals, and Office upload/export. No external model or provider is called. The workbook context reads use the local API and original fixture bytes.

For explicit full-document visual exports, validate_annotations runs inside the Orchestrator before the deterministic adapter exporter. Unit tests verify complete coverage, unchanged snapshot signatures, result reuse across approval/resume, invalidation after annotation mutation, and failure/incomplete/no-explicit-export gates. The UI maps a matching in-run Validator result into the existing Final Review panel and skips a duplicate provider request only when the selected model and final snapshot match; XLSX export remains on its separate workbook evidence path.

The separate live churn demo opens an 18-row synthetic Customers workbook with last_login stored as typed Excel dates and displayed as yyyy-mm-dd, plus a blank Churn Risk output column. The browser verifies all eighteen output cells remain blank, the dedicated High / Medium / Low task and Autopilot preset load, and an unconfigured run opens Settings without a provider request.

Manual packaged macOS desktop acceptance also opened the native folder chooser on an isolated QA build, selected a disposable fixture containing a nested PDF, confirmed the project listed nested/review/nested-contract.pdf, and opened it through the app's Tauri filesystem access; the viewer rendered all three PDF pages. The QA app bundle, fixture folder, and isolated app state were removed after the check. This verifies the picker grant and nested read path on this machine; it is not an automated picker test.

PDF native export now preserves the uploaded PDF page content and adds vector outlines and numbered markers. The focused adapter test reloads the result, confirms the original searchable text and content stream remain, checks the new rectangle geometry, and verifies the source bytes and SHA-256 are unchanged. Office and image sources retain the rendered-page PDF fallback.

Workspace export now recovers from an expired document session by reopening the already connected source and checking its SHA-256 before rebinding saved annotations. The production browser E2E forces one folder export to return HTTP 410, then verifies a single source upload, a new server document ID, and the persisted human-approved record in the final export. A separate provider-free run streams navigate_page and scroll_document activity to the real React viewer; the final result intentionally omits those events, and the browser asserts page 2, a positive scroll offset, the highlight overlay, and both activity rows. Its capture is output/playwright/ui-redesign/agent-navigation-stream-1440x900.png. While a later navigation request is held in flight, the same E2E asserts that the sticky latest-activity strip intersects the viewport and contains live event text; the capture is output/playwright/ui-redesign/agent-live-activity-1440x900.png. The same E2E leaves page 2 opened but uninspected in the first 12-page segment, verifies that the host resumes at page 2 rather than skipping it, and then reaches checked coverage on all 15 pages. Direct Agents SDK tests also verify that a full-scope call returns incomplete plus exact remainingPages after an early final answer, and that full-scope export remains disabled for an incomplete checkpoint. The browser also stages a page with a persistent converter warning, exposes a human acknowledgement after the page is reviewed, and verifies that the saved run and workbench status become complete; unit tests cover the matching image-only acknowledgement and correction of same-page review counts. Legacy one-page continuations recover full-document scope from saved coverage, and only explicit inspection checkpoints—not old visit history—restore pages as inspected. The process-level API test uses a local fake Responses endpoint for the real Task Planner and Agents SDK routes, pauses on a high-priority review, resumes the same RunState after approval, continues to page 3, and verifies approved plus automatic records in JSON export. It also holds the final model answer and proves tool events arrive through Express SSE before the result frame. npm run test:agent-sse-browser-e2e additionally builds and drives the production React UI in Chromium through the real Task Planner and Agents SDK Express SSE routes, using only a loopback fake Responses API. After streamed navigation, scrolling, and Human Review, it corrects the page-2 HIGH RISK suggestion to LOW RISK and explicitly accepts the generated rule for remaining pages. It verifies the request uses approved:false for the original candidate, resumes the same Run, annotates page 3 as LOW RISK, and exports both the human-corrected page-2 finding and automatic page-3 annotation as JSON. All model responses come from loopback fixtures; no external provider is called. Server security tests reject a DNS-rebinding-style matching arbitrary Host/Origin, allow configured deployment origins, and bound malformed upload-limit configuration to its finite default. Packaged Tauri startup now always launches its bundled API rather than trusting an unauthenticated health response from an existing listener on port 3001.

Full-document coverage is server-authoritative: the API derives page count from the uploaded session and stores verified inspection checkpoints against the original source hash and task scope in the encrypted session record. A client cannot unlock Validator/export by submitting alreadyInspectedPages; process-level API coverage rejects a false page count, rejects a forged complete list, and repeats the forgery after restarting the API to verify the encrypted checkpoint restores safely. A live Run compares the exact provider identity through a non-serialized, process-memory WeakMap and constant-time equality check; the identity is never written to the pending-run store or returned from the API.

Final checks passed: npm test (202/202), npm run lint, npm run build -- --configLoader native, npm run test:browser-e2e, npm run test:agent-sse-browser-e2e, npm run test:office-native-export-libreoffice, npm run prepare:desktop-runtime, npm run test:desktop-runtime (bundled aarch64 macOS API readiness and parent-pipe shutdown), cargo fmt --all -- --check, cargo check, npm audit --audit-level=high (0 vulnerabilities), and git diff --check.

final result: passed