test(chunking): Capture semantic chunking benchmark evidence - #414

Individual Eval Scores

PASS code-review code-review/async-for-each-invites: project invite sending uses forEach with an async callback and returns completed invite results for audit logging: green 1.00
PASS code-review code-review/eval-optional-assertion-rationale: eval judge reporting records failure reasons for optional should_find assertions even when the eval score passes: green 1.00
PASS code-review code-review/falsey-default-limit: search request building supports limit 0 as a metadata-only request but defaults falsey limits with ||: green 1.00
PASS code-review code-review/ignores-style-issues: functionally correct code with style issues, verbose conditionals, and magic numbers: green 1.00
PASS code-review code-review/pagination-floor-total-pages: fetchAllItems computes total pages for a paginated API where the final page can be partially full: green 1.00
PASS code-review code-review/permission-level-comparison: permission filtering uses a role hierarchy where higher roles include lower-role permissions: green 1.00
PASS code-review code-review/robots-prefix-blocks-public-metadata: a robots.txt golden test disallows a prefix that also blocks a documented public metadata endpoint: green 1.00
PASS code-review code-review/sentry-empty-repo-409-cache: Sentry source code integration repeatedly fetches a GitHub tree for an empty repository that returns 409: green 1.00
PASS code-review code-review/sentry-event-grouping-info-mutates-class-context: Sentry grouping info endpoint builds per-request grouping context from a class-level initial_context template: green 1.00
PASS code-review code-review/sentry-explorer-autofix-missing-group-id: Sentry explorer autofix metadata is used by Slack integration to continue an issue-specific flow: green 1.00
PASS code-review code-review/sentry-group-index-delete-undefined-group-list: Sentry group index delete helper handles request shapes where no branch assigns group_list: green 1.00
PASS code-review code-review/sentry-seer-race-condition-unbound-local: Sentry Seer grouping race-condition logging references new_has_group after conditional initialization: green 1.00
PASS code-review code-review/sentry-spans-null-attributes: Sentry span buffering receives spans with null attributes: green 1.00
PASS code-review code-review/sentry-user-details-revoke-staff-default-org: Sentry user details update handles a staff or superuser who is no longer in the default organization: green 1.00
PASS code-review code-review/sentry-vitest-evals-duration-sixty-seconds: GitHub reporter duration formatting rounds seconds at minute boundaries without carrying into the next unit: green 1.00
PASS code-review code-review/sentry-vitest-evals-github-reporter-positional-json: GitHub reporter CLI argument parsing initializes a default JSON report path before parsing arguments while also keeping a positional JSON path fallback: green 1.00
PASS code-review code-review/stale-autosave-closure: React autosave effect promises to save the latest editor text on an interval: green 1.00
PASS e2e eval-bug-detection/null-property-access: code that accesses properties on an array .find() result without null checking: green 1.00
PASS e2e eval-bug-detection/missing-await: async cache lookup missing await, causing a Promise object to be used as a truthy value: green 1.00
PASS e2e eval-bug-detection/wrong-comparison: permission check using <= instead of >=, inverting the access control logic: green 1.00
PASS e2e eval-bug-detection/stale-closure: React useEffect with setInterval that captures count in a stale closure: green 1.00
PASS e2e eval-precision/ignores-style-issues: functionally correct code with style issues (mixed naming conventions, verbose conditionals, magic numbers): green 1.00
PASS e2e eval-security-scanning/sql-injection: SQL query built via string interpolation with user-supplied search parameters: green 1.00
PASS e2e eval-security-scanning/xss-reflected: HTML template that renders URL query parameter directly into page without escaping: green 1.00
PASS security-review security-review/sentry-autofix-settings-get-project-access: Autofix automation settings GET lists every project in the organization instead of using OrganizationEndpoint.get_projects: green 1.00
PASS security-review security-review/sentry-group-search-view-visit-visibility: group search view visit endpoint updates private views by ID without applying the existing object permission or visibility checks: green 1.00
PASS security-review security-review/sentry-preprod-snapshot-project-access: preprod snapshot endpoint fetches artifacts by organization only and returns or deletes project artifacts without checking project membership: green 1.00
PASS security-review security-review/sentry-release-threshold-empty-project-filter: release threshold endpoint builds an empty ORM filter when the caller has no accessible projects: green 1.00
PASS security-review security-review/sentry-replay-count-project-scope-overwrite: replay count helper replaces request-scoped projects with projects from issue IDs resolved across the organization: green 1.00
PASS security-review security-review/sentry-replay-delete-read-scope: replay detail endpoint grants DELETE to project:read and then enqueues destructive replay deletion: green 1.00
PASS security-review security-review/sentry-slack-options-load-unscoped-group: Slack options-load webhook validates Slack but resolves a group by caller-controlled ID without binding it to the Slack workspace or organization: green 1.00
PASS security-review security-review/sentry-workflow-connect-workflows-authz: detector-to-workflow connection validates target workflows exist in the organization but does not check permission to modify those workflows: green 1.00
PASS security-review security-review/sentry-workflow-open-periods-project-access: workflow open-period endpoint resolves detectorId or groupId by organization only and returns issue activity without project permission checks: green 1.00
PASS verification security-review/preprod-size-analysis-base-artifact-keep: verifier keeps a baseArtifactId authorization finding when the changed code checks organization_id but never checks project access: green 1.00
PASS verification security-review/slack-signature-stub-reject: verifier rejects a false positive that treats the fixture's Slack validation stub as the vulnerability: green 1.00
PASS verification security-review/workflow-open-periods-project-access-keep: verifier keeps an open-periods authorization finding when detectorId or groupId is org-scoped but project access is never checked: green 1.00
FAIL code-review code-review/sentry-action-dedup-by-workflow: Sentry workflow engine evaluates notification actions from different workflows that target the same destination and issue: red 0.00

• should_find[0] not met: No findings were reported, so the deduplication key issue was not detected.
  FAIL code-review code-review/sentry-ai-conversation-multipart-message: Sentry AI conversation rendering receives a message made of multiple content parts: red 0.00
• should_find[0] not met: The agent reported no findings, so the required issue about multi-part AI conversation messages not being rendered or parsed correctly was not detected.
  FAIL code-review code-review/sentry-code-review-pr-closed-trigger-filter: Sentry code review webhook receives a pr_closed event for a repo with enabled_code_review true and an empty code_review_triggers list: red 0.00
• should_find[0] not met: No findings were reported, so the expected behavior about pr_closed bypassing the trigger gate when code_review_triggers is empty was not detected.
  FAIL code-review code-review/sentry-cursor-service-account-api-key: Sentry Cursor integration verifies an API key created for a Cursor service account: red 0.00
• should_find[0] not met: No findings were reported, so the issue with verify_api_key relying only on /v0/me endpoint failing for Cursor service account keys was not detected.
  FAIL code-review code-review/sentry-dashboard-axis-range-existing-widget: Sentry dashboard widget builder opens an existing widget saved before axisRange was introduced: red 0.00
• should_find[0] not met: No findings were reported, so the required issue about convertWidgetToBuilderStateParams leaving axisRange undefined was not detected.
  FAIL code-review code-review/sentry-dashboard-delete-side-nav-stale: Sentry dashboards landing page deletes a starred dashboard shown in the side navigation: red 0.00
• should_find[0] not met: No findings were reported, so the required bug about deleted dashboards not updating the side navigation was not detected.
  FAIL code-review code-review/sentry-dashboard-other-series-color: Sentry dashboard new timeseries visualization renders the Other series returned from top-N grouping: red 0.00
• should_find[0] not met: No findings were reported, so the required issue about Other series not being mapped to neutral grey color was not detected.
  FAIL code-review code-review/sentry-detector-validator-user-context: Sentry detector endpoints build validator context for detector type feature flag checks targeted at the requesting actor: red 0.00
• should_find[0] not met: No findings were reported, so the issue of get_detector_validator omitting request.user from the validator context was not detected.
  FAIL code-review code-review/sentry-fixability-missing-issue-summary: Sentry fixability calculation calls Seer with cached issue summaries available: red 0.00
• should_find[0] not met: No findings were reported, so the issue about get_and_update_group_fixability_score not consistently passing issue summaries to the fixability endpoint was not detected.
  FAIL code-review code-review/sentry-group-events-snuba-errors: Sentry group events endpoint receives a Snuba timeout or query error: red 0.00
• should_find[0] not met: No findings were reported, so the missing handle_query_errors issue was not detected.
  FAIL code-review code-review/sentry-issue-highlights-missing-transaction-default: Sentry issue highlight defaults should include transaction context for issue displays: red 0.00
• should_find[0] not met: The agent reported no findings, so the required issue about default highlight field set omitting transaction context was not detected.
  FAIL code-review code-review/sentry-metric-issue-chart-open-period: Sentry metric issue chart loads an open period that can change the selected time range: red 0.00
• should_find[0] not met: No findings were reported, so the race condition between metric issue chart data requests and open period loading was not detected.
  FAIL code-review code-review/sentry-metric-issue-open-period-start: Sentry metric issue sidebar queries contributing issues for a priority change event during an open period: red 0.00
• should_find[0] not met: No findings were reported, so the required bug about metricDetectorTriggeredSection using event date instead of open period start was not detected.
  FAIL code-review code-review/sentry-semver-build-ordering: Sentry resolves the next release among semver releases that share the same version but differ by build metadata: red 0.00
• should_find[0] not met: No findings were reported, so the semver build metadata ordering issue was not detected.
  FAIL code-review code-review/sentry-similar-issues-duplicate-score: Sentry similar issues formatting receives multiple parent hashes for the same group sorted by best similarity first: red 0.00
• should_find[0] not met: Neither finding addresses the issue of duplicate parent_group_id values causing group_data to be overwritten, resulting in the best-ranked duplicate retaining its position but displaying the worse similarity score.
  FAIL code-review code-review/sentry-trace-metrics-none-unit: Sentry trace metric requests use the explicit unit value none to select spans with no metric unit set: red 0.00
• should_find[0] not met: The agent found an issue with empty-string metric_unit being treated like None, but the expected finding is specifically about metric_unit='none' being used as a literal string filter instead of representing 'no unit'.
  FAIL code-review code-review/sentry-user-misery-eap-validation: Sentry metric alert validation runs user_misery alerts through the EAP function allowlist: red 0.00
• should_find[0] not met: The agent's finding is about empty string vs None for zero-argument functions, not about user_misery being missing from the EAP function allowlist.
  FAIL e2e eval-bug-detection/off-by-one: pagination logic that uses Math.floor instead of Math.ceil, skipping the last page: red 0.00
• should_find[0] severity mismatch: expected medium, got high
  FAIL security-review security-review/sentry-preprod-size-analysis-base-artifact-access: public size-analysis endpoint accepts a baseArtifactId and loads the base artifact by organization only: red 0.00
• should_find[0] not met: No findings were reported, so the authorization bypass via baseArtifactId lacking project access check was not detected.