feat(snapshot): bulk snapshot endpoints for OpenGRC (#7505)

[Alexandre Gaspard (alexgaspard)]

Proposed changes

• Two new engine indexes holding the current verified state of every endpoint: snapshot-attack-observation (one document per tenant, endpoint, technique, expectation type and scenario) and snapshot-vulnerability-observation (one document per tenant, endpoint, finding type and finding value). Both are built by keyset paged handlers.

• Keyset paging in the indexing loop, and cursor paged search in the engine abstraction, for both Elasticsearch and OpenSearch.

• Two endpoints for OpenGRC to read those indexes differentially:

  • POST /api/tenants/{tenantId}/snapshot/attack-observations/search
  • POST /api/tenants/{tenantId}/snapshot/vulnerability-observations/search

  Each response carries the page plus snapshot_window_end, indexed_through and snapshot_ready, so a client can tell whether the inventory it just read was complete.

• New read only capability ACCESS_SNAPSHOT_OBSERVATION, with its nine translations.

• Everything is behind the BULK_SNAPSHOT_EXPORT preview flag, off by default. With the flag off the handlers are not registered, no index is created and both endpoints return 404.

• Documentation in docs/reference/apis/bulk-snapshot.md, plus the capability row in users-and-rbac.md.

One additive migration: nullable indexing_status_last_id column on indexing_status, no backfill, no cursor reset. No new configuration key.

Where to start

The diff is large but most of it is tests and generated types. The parts worth reading closely:

• SnapshotObservationService for the window computation: safety lag, clamping, readiness.
• AttackObservationRepository and VulnerabilityObservationRepository for the two native queries that define the grains.
• SnapshotObservationApi and SnapshotObservationApiTest for RBAC, tenant isolation and cursor validation.

Testing Instructions

1. Start the services:
   cd openaev-dev && docker compose up -d openaev-dev-pgsql openaev-dev-minio openaev-dev-elasticsearch openaev-dev-rabbitmq
2. API layer:

   mvn -pl openaev-api -am test -Dtest='SnapshotObservationApiTest,SnapshotObservationFeatureFlagApiTest,SnapshotObservationServiceTest,SnapshotObservationMapperTest,SnapshotCursorCodecTest' -Dsurefire.failIfNoSpecifiedTests=false -Djacoco.skip=true
   

3. Indexing and regressions (the conditional handlers change EngineContext.getModels()):

   mvn -pl openaev-api -am test -Dtest='AttackObservationIndexingTest,VulnerabilityObservationIndexingTest,AttackObservationIndexingFlagOffTest,VulnerabilityObservationIndexingFlagOffTest,CursorPagedSearchTest,BulkSnapshotExportConditionTest,IndexingRegressionTest,IndexingRegressionIntegrationTest,IndexingTenantScopeRegressionTest,TenantStatementInspectorTest' -Dsurefire.failIfNoSpecifiedTests=false -Djacoco.skip=true
   

4. Boot with openaev.enabled-dev-features=BULK_SNAPSHOT_EXPORT and check that both indexes are created and that both endpoints answer for a non admin user holding the capability. Boot without the flag and check that neither the indexes nor the endpoints exist.

Related issues

• Closes feat(snapshot): bulk snapshot endpoints for OpenGRC #7505

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality
• I wrote test cases for the relevant uses case
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality
• For bug fix -> I implemented a test that covers the bug

Further comments

A few choices worth confirming:

• Only findings on ENDPOINT assets are exported, consistently with EsVulnerableEndpoint. Findings on other asset types are skipped.
• base_dependencies holds the asset only. The simulation and the scenario are carried as ACL restrictions, so deleting a simulation does not orphan a document.
• last_verified_at is the latest finding_updated_at of the grain rather than the inject execution date, because it is the only timestamp every producer sets.
• indexed_through is stored per stream and not per tenant, so snapshot_ready reflects the platform backlog: a quiet tenant can be held back by a noisy one. Documented on the page as a known limitation.
• The attack stream only contains observations produced inside a scenario. The vulnerability stream also exports findings produced outside any scenario, with a null last_scenario_id.

[github-actions]

📖 Documentation check — ✅ Passed

44 functional file(s), 3 doc file(s) changed.

Documentation-worthy changes detected and documentation was updated. 👏

Detected changes (covered by doc updates)

• 🔴 New REST API controller — openaev-api/src/main/java/io/openaev/api/snapshot/SnapshotObservationApi.java
• 🟡 New preview feature flag — openaev-api/src/main/java/io/openaev/rest/settings/PreviewFeature.java

[github-actions]

✅ Container vulnerability scan — Passed

Previously reported findings are no longer present.

Image	Critical	High	Total	Status
Standard	0	0	0	✅ Clear
UBI9	0	0	0	✅ Clear

View workflow run · Standard JSON report · UBI9 JSON report

_{Updated from CI run attempt 1.}

[github-actions]

Thank you for your contribution. This PR is but one step away from being ready for merging: all commits must be PGP-signed. To get started, please see https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

[Copilot]

Pull request overview

Adds a new “bulk snapshot export” capability for OpenGRC-style integrations by introducing two new snapshot observation streams (attack + vulnerability) backed by dedicated engine indexes, plus cursor-paged read APIs that expose the current verified endpoint state with explicit window/horizon metadata. It also extends the indexing loop and engine abstraction to support keyset paging and cursor-paged search (Elastic + OpenSearch), and wires RBAC/capability gating and preview-flag gating end-to-end.

Changes:

• Introduces two new indexable snapshot models (attack/vulnerability observations) with keyset-paged indexing handlers and native-query grains.
• Adds cursor-paged search support in both ElasticService and OpenSearchService (keyset predicate + strict tenant/grant restrictions).
• Adds preview-flag gating (BULK_SNAPSHOT_EXPORT), a new tenant capability (ACCESS_SNAPSHOT_OBSERVATION), API endpoints, tests, and documentation.

Reviewed changes

Copilot reviewed 65 out of 66 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
openaev-model/src/main/java/io/openaev/utils/OpenSearchUtils.java	Adds keyset “resume after (ts,id)” predicate builder for cursor paging.
openaev-model/src/main/java/io/openaev/utils/ElasticUtils.java	Adds Elastic equivalent keyset predicate builder.
openaev-model/src/main/java/io/openaev/service/OpenSearchService.java	Persists keyset cursor (timestamp + lastId) and adds cursor-paged search implementation (OpenSearch).
openaev-model/src/main/java/io/openaev/service/IndexingCursor.java	New record to represent persisted indexing cursor (timestamp + optional lastId).
openaev-model/src/main/java/io/openaev/service/EsIndexingUtils.java	Adds keyset cursor computation + grace-window capping logic for (ts,id).
openaev-model/src/main/java/io/openaev/service/ElasticService.java	Persists keyset cursor (timestamp + lastId) and adds cursor-paged search implementation (Elasticsearch).
openaev-model/src/main/java/io/openaev/engine/model/snapshotobservation/VulnerabilityObservationHandler.java	New keyset-paged handler to fetch vulnerability observation grains for indexing.
openaev-model/src/main/java/io/openaev/engine/model/snapshotobservation/EsVulnerabilityObservation.java	New indexable ES model for vulnerability observation snapshot docs.
openaev-model/src/main/java/io/openaev/engine/model/snapshotobservation/EsAttackObservation.java	New indexable ES model for attack observation snapshot docs.
openaev-model/src/main/java/io/openaev/engine/model/snapshotobservation/AttackObservationHandler.java	New keyset-paged handler to fetch attack observation grains for indexing.
openaev-model/src/main/java/io/openaev/engine/model/EsBase.java	Registers new snapshot observation discriminators for engine polymorphic schema.
openaev-model/src/main/java/io/openaev/engine/Handler.java	Extends handler contract with optional keyset-paged fetch(from, fromId, limit) + isKeysetPaged().
openaev-model/src/main/java/io/openaev/engine/EngineService.java	Adds cursor-paged search API and max page size constant.
openaev-model/src/main/java/io/openaev/engine/BulkSnapshotExportCondition.java	New Spring condition to gate handler bean registration behind preview feature flag.
openaev-model/src/main/java/io/openaev/engine/api/CursorPageQuery.java	New record describing cursor-paged query bounds (since/after/windowEnd/size).
openaev-model/src/main/java/io/openaev/database/repository/VulnerabilityObservationRepository.java	Adds native keyset-grain query + existence probe for vulnerability observations.
openaev-model/src/main/java/io/openaev/database/repository/AttackObservationRepository.java	Adds native keyset-grain query + existence probe for attack observations.
openaev-model/src/main/java/io/openaev/database/raw/RawVulnerabilityObservationIndexing.java	New projection interface for vulnerability observation indexing query.
openaev-model/src/main/java/io/openaev/database/raw/RawAttackObservationIndexing.java	New projection interface for attack observation indexing query.
openaev-model/src/main/java/io/openaev/database/model/ResourceType.java	Adds SNAPSHOT_OBSERVATION resource type.
openaev-model/src/main/java/io/openaev/database/model/IndexingStatus.java	Adds persisted indexing_status_last_id field to support keyset cursor.
openaev-model/src/main/java/io/openaev/database/model/Capability.java	Adds ACCESS_SNAPSHOT_OBSERVATION capability and removes credential helper method.
openaev-front/src/utils/lang/zh.json	Adds translation for ACCESS_SNAPSHOT_OBSERVATION.
openaev-front/src/utils/lang/ru.json	Adds translation for ACCESS_SNAPSHOT_OBSERVATION.
openaev-front/src/utils/lang/ko.json	Adds translation for ACCESS_SNAPSHOT_OBSERVATION.
openaev-front/src/utils/lang/ja.json	Adds translation for ACCESS_SNAPSHOT_OBSERVATION.
openaev-front/src/utils/lang/it.json	Adds translation for ACCESS_SNAPSHOT_OBSERVATION.
openaev-front/src/utils/lang/fr.json	Adds translation for ACCESS_SNAPSHOT_OBSERVATION.
openaev-front/src/utils/lang/es.json	Adds translation for ACCESS_SNAPSHOT_OBSERVATION.
openaev-front/src/utils/lang/en.json	Adds translation for ACCESS_SNAPSHOT_OBSERVATION.
openaev-front/src/utils/lang/de.json	Adds translation for ACCESS_SNAPSHOT_OBSERVATION.
openaev-front/src/utils/api-types.d.ts	Updates generated API types for snapshot models/DTOs and preview feature/capability enums.
openaev-api/src/test/java/io/openaev/utilstest/OpenSearchUtilsTest.java	Adds unit test for OpenSearch keyset predicate shape.
openaev-api/src/test/java/io/openaev/utilstest/ElasticUtilsTest.java	Adds unit test for Elastic keyset predicate shape.
openaev-api/src/test/java/io/openaev/service/PermissionServiceTest.java	Adds permission regression tests for SNAPSHOT_OBSERVATION SEARCH gating.
openaev-api/src/test/java/io/openaev/service/EsIndexingUtilsTest.java	Adds unit tests for keyset cursor computation and grace-window behavior.
openaev-api/src/test/java/io/openaev/engine/VulnerabilityObservationIndexingFlagOffTest.java	Verifies handler/model not registered when preview flag is off.
openaev-api/src/test/java/io/openaev/engine/IndexingRegressionIntegrationTest.java	Regression coverage for keyset cursor additive slice (null lastId for existing models, etc.).
openaev-api/src/test/java/io/openaev/engine/CursorPagedSearchResultWindowTest.java	Proves cursor paging works past engine max-result-window using small cap.
openaev-api/src/test/java/io/openaev/engine/BulkSnapshotExportConditionTest.java	Unit tests for feature-flag parsing + anti-drift with PreviewFeature enum.
openaev-api/src/test/java/io/openaev/engine/AttackObservationIndexingFlagOffTest.java	Verifies handler/model not registered when preview flag is off.
openaev-api/src/test/java/io/openaev/architecture/TenantScopedEntrypointsTxCtxArchTest.java	Registers new snapshot API entrypoints as TxCtx-required.
openaev-api/src/test/java/io/openaev/api/snapshot/SnapshotObservationServiceTest.java	Unit tests for horizon/window computation, clamping, paging semantics.
openaev-api/src/test/java/io/openaev/api/snapshot/SnapshotObservationMapperTest.java	Verifies exact wire contract field mapping/serialization for both streams.
openaev-api/src/test/java/io/openaev/api/snapshot/SnapshotObservationFeatureFlagApiTest.java	Ensures endpoints return 404 when preview flag is off even with capability.
openaev-api/src/test/java/io/openaev/api/snapshot/SnapshotCursorCodecTest.java	Unit tests for cursor encoding/decoding, tenant binding, and rejections.
openaev-api/src/test/java/io/openaev/api/capabilities/CapabilityTreeBuilderTest.java	Ensures snapshot capability appears/vanishes based on preview gate; platform vs tenant scope.
openaev-api/src/main/java/io/openaev/rest/settings/PreviewFeature.java	Adds BULK_SNAPSHOT_EXPORT preview feature enum entry.
openaev-api/src/main/java/io/openaev/migration/V6_20260819100000000__Indexing_status_keyset_cursor.java	Adds nullable DB column indexing_status_last_id (Flyway).
openaev-api/src/main/java/io/openaev/api/snapshot/SnapshotObservationService.java	Implements window/horizon computation + delegates to cursor-paged engine search.
openaev-api/src/main/java/io/openaev/api/snapshot/SnapshotObservationMapper.java	Maps engine docs to output DTOs for both streams.
openaev-api/src/main/java/io/openaev/api/snapshot/SnapshotObservationApi.java	Adds two tenant-scoped snapshot search endpoints gated by preview flag + capability.
openaev-api/src/main/java/io/openaev/api/snapshot/SnapshotCursorCodec.java	Implements opaque, URL-safe base64 cursor codec with tenant binding and validation.
openaev-api/src/main/java/io/openaev/api/snapshot/form/VulnerabilityObservationOutput.java	New output DTO for vulnerability observation stream.
openaev-api/src/main/java/io/openaev/api/snapshot/form/SnapshotSearchOutput.java	New response envelope carrying page + window/horizon/readiness metadata.
openaev-api/src/main/java/io/openaev/api/snapshot/form/SnapshotSearchInput.java	New input DTO supporting since/cursor, clamped page size and safety lag.
openaev-api/src/main/java/io/openaev/api/snapshot/form/AttackObservationOutput.java	New output DTO for attack observation stream.
openaev-api/src/main/java/io/openaev/api/capabilities/CapabilityTreeBuilder.java	Generalizes capability preview-gating (credentials + snapshot) and caches “all gates enabled” trees.
openaev-api/src/main/java/io/openaev/api/capabilities/CapabilityApi.java	Builds capability tree based on enabled preview features set (not just credentials).
docs/mkdocs.yml	Adds Bulk snapshot export doc page to nav.
docs/docs/reference/apis/bulk-snapshot.md	New API reference describing cursor semantics, readiness/window, and client obligations.
docs/docs/administration/users-and-rbac.md	Documents the new “Access observation snapshots” capability.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.