Feature/voxel annotation - #858
Open
briossant wants to merge 252 commits into
Open
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Collaborator
|
Can you complete the CLA? |
jbms
reviewed
Nov 3, 2025
jbms
reviewed
Nov 3, 2025
jbms
reviewed
Nov 3, 2025
Collaborator
|
The brush hover outline (circle where the mouse pointer is) seems to go away in some cases when changing the zoom level. |
briossant
force-pushed
the
feature/voxel-annotation
branch
from
November 10, 2025 09:51
2d5359d to
9d15526
Compare
Author
|
|
- Introduced a new dummy `MultiscaleVolumeChunkSource`. - Added `VoxelAnnotationRenderLayer` for voxel annotation rendering. - Implemented `VoxUserLayer` with dummy data source and rendering. - Added tools and logs for voxel layer interactions and debugging. - Documented voxel annotation specification and implementation details.
- Added a backend `VoxDummyChunkSource` that generates a checkerboard pattern for voxel annotations. - Implemented frontend `VoxDummyChunkSource` with RPC pairing to the backend. - Updated documentation with details on chunk source architecture and implementation.
…t seems there is some fighting.
…s to corruped the chunk after the usage of the tool. Added a front end buffer which is the only drawing storage for now. Added user settings to set the voxel_annotation layer scale and bounds. Added a second empty source to DummyMultiscaleVolumeChunkSource to prevent crashs when zoomed out too much
…lobal one (there where a missing convertion) ; add a primitive brush tool
…umeChunkSource and update related imports
…panded brush settings
…ation, and improved backend edit handling
…map options and UI settings
…r remote workflows, label creation, and new drawing tools
…enhanced UI rendering - Add label management with ID generation, selection, and default initialization. - Persist labels using IndexedDB with dedicated key management. - Integrate label creation and selection into the UI with real-time updates. - Update shaders for Uint64 handling and segment color hashing. - Modify voxel initialization to support deterministic scale keys.
- Replace IndexedDB-based label management with RPC-driven persistence. - Add `getLabelIds` and `setLabelIds` methods for backend label synchronization. - Introduce new RPC handlers for label retrieval and updates. - Update frontend to initialize voxel map and manage labels via RPC calls. - Consolidate label persistence logic in `VoxSource` and its subclasses. - Remove debugging logs and redundant IndexedDB related code.
… and multiscale parents Adds four end-to-end tests against the S3 (MinIO) zarr pipeline: repainting over already-written chunks, repaint with a gzip codec, dense high-bit uint64 labels, and a multiscale OME dataset verifying a non-empty downsampled parent after the cascade.
…s all dispatched strokes The swap-on-arrival overlay clear fired as soon as a refetched real chunk reached the GPU, regardless of whether the write behind that refetch included every stroke the overlay represents. Painting a second stroke across chunks of a still-flushing first stroke made the second stroke vanish until its own flush landed (~1s), because the first stroke's reload cleared the shared overlay chunk. Every dispatched operation (brush, flood fill) now carries a monotonic frontend seq, threaded through pendingEdits. On flush, the backend records per chunk the max seq durably written (lastFlushedSeq) and echoes it in the existing reload message (coveredSeqs). The frontend tracks, per overlay chunk, the seq of the last dispatch whose preview touched it (lastDispatchedSeq) plus the keys of the in-progress stroke (undispatchedPreviewKeys), and clears an overlay chunk only when the arriving data covers its last dispatched stroke and no stroke is being painted on it. A skipped clear is always re-armed by the covering write's own reload. Reload RPCs also drain the frontend's pending chunk-update queue before arming listeners, so a stale refetch already queued cannot trigger a freshly armed clear.
…ck full downsample steps Two fixes for overlay clears and data integrity at LOD >= 1: - The coverage a cascade reload claims for the origin overlay is now captured once per chain, before the first child read, instead of at each step: a flush completing mid-chain is not present in the data the chain propagates, and claiming it cleared the overlay over a parent that lacked those edits. - The child read and parent-update computation now run inside withChunkLock(parentKey) along with the write. With only the write serialized, two concurrent chains targeting the same parent could land in the wrong order and durably overwrite the fresher parent with a result computed from a stale read; inside the lock, the later writer computed from the later read.
…oll back undispatched strokes The coverage guard tracked which strokes touched which overlay chunks in controller-side prediction state (undispatchedPreviewKeys / lastDispatchedSeq), populated by previews and consumed at dispatch. Any path where a previewed stroke never dispatched left that state stuck and permanently blocked the overlay chunk from clearing: withCost silently dropping the dispatch at the stamina cap or on permission refusal, stopDrawing bailing on an empty stroke or a vanished editing context, and preview/backend divergence on filtered brushes. The stroke seq is now allocated before the first preview (beginStroke, snapshotted into activeStroke) and tags the overlay chunks directly on InMemoryVolumeChunkSource as they are painted, so what the overlay shows and what the dispatch carries cannot diverge. The clear guard reduces to a single fire-time condition: coveredSeq >= overlay chunk's tag (an in-progress stroke's chunks are protected by construction, their tag is not yet covered by any write). Tags are purged with the chunk in deleteChunk, so the map only ever tracks live overlay chunks. Strokes whose edits will never be written are now rolled back explicitly (rollbackStroke) instead of waited for: the brush dispatch wrapper rolls back when withCost does not run the dispatch or the RPC fails, stopDrawing rolls back on its bail-out paths via the snapshotted context, and flood fill rolls back through the same mechanism (dropping the now-redundant affectedKeys tracking). The flood-fill fast path for unloaded chunks now carries a seq as well, so its write advances coverage like any other.
…ty on fresh-chunk listeners Voxel reload RPCs drained the entire pending chunk-update queue synchronously (flushPendingChunkUpdates) before arming fresh-chunk listeners, so that a stale refetch already queued could not trigger a freshly armed listener with pre-write data. That converted the queue's time-budgeted GPU upload work into synchronous stalls on every flush and cascade reload — a global remedy for a per-key ordering concern. Chunk updates are now stamped with a monotonic receipt seq as they arrive from the worker (queued or immediate). onNextFreshChunk records the seq current at arming, and firing only triggers listeners armed before the update that delivered the data; later-armed listeners stay armed for the next arrival. pendingFreshChunkGpu keeps the receipt of the update.new that delivered the data (not a later promotion update), since staleness is decided by when data was received, not when it reached the GPU. The drain in callChunkReload is removed: same causality guarantee, zero synchronous work at reload receipt, and the invariant no longer depends on queue-batching internals.
…oving voxel hooks from the chunk manager The swap-on-arrival mechanism lived inside ChunkQueueManager / ChunkSource: per-key fresh-chunk listeners, a pending-GPU arming map and an update-receipt seq stamped on every chunk update, plus firing logic in applyChunkUpdate (~110 lines in the core chunk machinery). Core chunk logic is the last place a feature branch should carry code: the test suite's coverage cannot vouch for side effects there, and every touched line raises the review bar for merging. All of it is replaced by observation through existing public API. The controller keeps a pending-swap map (one entry per real chunk, overwritten by newer reloads) and scans it on visibleChunksChanged: an `update.new` always builds a fresh Chunk object, so object identity against the chunk recorded at arming detects the swap, and state === GPU_MEMORY detects display. The coverage guard is unchanged and still read at swap time. chunk_manager's total branch footprint is back to a single additive method — invalidateChunks with the lazy option (keep the frontend chunk on display while the backend cache is invalidated and refetched) — with zero upstream lines modified. The single-tick texture handover moves to a VolumeChunkSource.addChunk override, which also fixes a latent leak: plain chunks.set over an existing GPU-resident chunk orphaned its texture. Known, accepted trade-off (documented at the arming site): without receipt stamping, a pre-write refetch still queued at arming time can resolve a swap with stale data. The window needs the update queue to lag behind RPC processing and heals within one round trip thanks to the backend cancelling in-flight downloads on write — a rare transient flicker in exchange for keeping the core hook-free.
- Exclude failed keys from the flush's real-chunk reload: their store was not modified, so reloading refetched unchanged data and raced with the failure rollback. - Fix the downsample benchmark's downsampleStep call, left at the old single-argument arity behind an `as any` cast; it no longer represented the real cascade. - Deduplicate the RPC record guards in the VOX_RELOAD_CHUNKS handler (asRecordOrUndefined) and the two processBackendEdits call sites in performBrush.
…ne flushed-seq entries Undo removes strokes from the store, so the coverage guard could never clear their overlays (no future data "contains" them): a quick undo after painting left a ghost stroke on screen until reload. Undone chunks now get an explicit overlay clear, like the write-failure rollback; the purged tags also neutralize the cascade reloads that follow. lastFlushedSeq entries are pruned once no cascade chain for their key is queued or running (the chain-start capture is the only reader that early pruning would break; a future flush recomputes a higher max from the globally monotonic dispatch seqs). The map is now bounded by cascades in flight instead of growing with every chunk ever edited. Also documents the accepted downsample-lock trade-off: the compute serializes with the write because applyEdits' network I/O dominates the lock anyway, and cross-parent parallelism is unaffected.
… discrete operations A context load resolving after a write+invalidate re-cached pre-write data unconditionally, poisoning the accessor's per-voxel cache until the next write or LRU eviction: a flood fill or locked erase started right after a stroke could then walk stale values (e.g. fill straight through a fresh stroke). invalidate now bumps a per-key load generation instead of dropping the pending load; the load loops and reloads when its generation moved, so callers keep sharing one self-correcting promise. Ordering operations alone could not fix this: downsample cascades stay deliberately concurrent with flushes. Discrete operations (undo/redo, flood fill) now start with flushBefore: wait out the in-flight flush (now tracked in currentFlush), cancel the debounce timer and flush what is still pending. Undo previously called flushPending directly, which misses a flush already in flight — undoing right after two strokes could pop the first stroke's action while the second was mid-flush, reverting shared voxels and corrupting the stack. Undo now targets the latest stroke, and a fill launched within the debounce window sees the stroke it follows. The brush path stays free-running (ordered by seq coverage).
…sion chain flushBefore ordered a discrete operation after the edits committed before it, but nothing ordered edits after an operation in progress: a debounced flush could fire during an undo's sequential per-chunk writes (or two rapid undos could overlap), and two concurrent applyEdits on the same chunk are isolated read-modify-writes — the last one silently drops the other's voxels, leaving a half-corrupted stroke and capturing oldValues out of order. All LOD-0 writers now run through one promise chain (runExclusive): the debounced flush, undo/redo (whole body, including the pending flush), and flood fill's flush-first step. Brush commits stay a synchronous append; only their flush enters the chain. Cascades stay outside: they write LOD >= 1 under per-parent locks, their LOD-0 reads self-correct via the load generations, and the last LOD-0 writer enqueues its own cascade. Replaces currentFlush/flushBefore. Also documents the accepted partial-undo limitation: chunks reverted before a mid-undo failure stay reverted while the action returns to the stack.
…aring them immediately Undo cleared overlays at once, revealing the pre-stroke chunk on the GPU; a refetch of the just-flushed stroke already sent by the worker then landed on top, making the undone stroke blink back for a round trip before the reverted data arrived. Near-deterministic when undoing right after painting, since flush-first guarantees a fresh write (and its refetch) precedes every undo. Undo/redo now emit a single reload marked isRollback: the frontend purges the overlay tags (an undone stroke's tag can never be covered by a future write) and arms a normal swap, so the overlay keeps showing the stroke until real data arrives and every revealed state is consistent with the previous one. Known edge: Ctrl+Z mid-brush-drag drops the in-progress preview on the rolled-back chunks until its dispatch rewrites it.
Author
|
Hey, I just made many fixes and rework to the drawing pipeline, the most noticeable thing is that now the chunks are reloaded when the data actually arrives, not after a small delay. The conclusion of those 16 commits is that there should no longer be any flickering when drawing ("any" might be too strong, we'll see with usage 😅 ). I still have a few reviews to tackle, I'll restart the work on this now. |
Contributor
This new system generally seems really good! Much nicer to use :) |
jbms
reviewed
Aug 10, 2026
seankmartin
reviewed
Aug 11, 2026
seankmartin
left a comment
seankmartin
left a comment
Contributor
There was a problem hiding this comment.
Still looking through, but wanted to bring up what I found so far. Also some tests seem to be failing
| ); | ||
| } | ||
|
|
||
| async write(key: string, value: ArrayBuffer): Promise<void> { |
Contributor
There was a problem hiding this comment.
I haven't thought through this in full, but somehow the enable writing checkbox in the UI feels shown too often.
Right now I think it will show on any s3 + zarr combination without any restriction. That relates to what I mentioned about the sharding. But also on public readonly buckets, or data that is not rank 3. It seems it could be useful if we could precondition this showing in the UI a bit more.
So far there would be (that I can think of)
- The zarr codec
- The rank of the subsource
- The bucket permissions, which seems the hardest to check. Maybe if we improve the UI a bit more this would be ok to have this third one still show. Think the current is quite unclear given its going to show on every s3 + zarr combo after this
Not sure if @jbms or @chrisj have thoughts or suggestions here
Author
There was a problem hiding this comment.
For those constraints that may be confusing, we could gray out the checkbox (instead of hiding it) with an indication of why when we hover it?
Contributor
|
Also now that we have s3 write and delete I wonder should the s3 common read/write/delete all use |
briossant
added a commit
to briossant/neuroglancer
that referenced
this pull request
Aug 15, 2026
- Rename PreviewMultiscaleChunkSource.ts and staminaCalibration.benchmark.ts to snake_case per repo convention. - Import #src/voxel_annotation/backend.js (not .ts) in chunk_worker.bundle.js. - Drop the unused _region parameter of updateFromCpuData. - S3 CORS docs: recommend explicit AllowedOrigins instead of "*" when enabling PUT/DELETE. - Rename DataSubsource.isPotentiallyWritable to supportsWriting and drop the UI-checkbox mention from its contract comment.
Author
Failing tests were an oversight on my part, should be good now. |
…s write coverage Operations now resolve with the vox chunk keys their write covers; the frontend drops any overlay chunk the preview tagged that is not in this set. A flood-fill or filtered-brush preview that overfilled on data the frontend lacked previously left ghost overlay chunks forever, since only backend-written chunks were ever reloaded. Brush voxels skipped because the store already holds the value count as covered: clearing their overlay would flash pre-write data until the prior write's own reload lands.
New layer JSON key "floodFillMorphological" (default true). When disabled the backend fill skips the channel-thickness gating and runs a plain 4-connected walk, matching the frontend preview's logic.
… edit path applyEdits now edits an isolated chunk passed to writeChunk and invalidates the shared cache entry instead of mutating it in place (08e0c65). Mock the queueManager's invalidateCachedChunks, assert on the chunk captured by the writeChunk spy, and give resident chunks a state <= SYSTEM_MEMORY_WORKER so their data is picked up.
- Rename PreviewMultiscaleChunkSource.ts and staminaCalibration.benchmark.ts to snake_case per repo convention. - Import #src/voxel_annotation/backend.js (not .ts) in chunk_worker.bundle.js. - Drop the unused _region parameter of updateFromCpuData. - S3 CORS docs: recommend explicit AllowedOrigins instead of "*" when enabling PUT/DELETE. - Rename DataSubsource.isPotentiallyWritable to supportsWriting and drop the UI-checkbox mention from its contract comment.
…writeChunk The 32-line chunk-key encoding (key-encoding prefix, physical-to-logical dimension permutation, read-chunk-to-chunk-shape division, separator join) was duplicated between download and writeChunk. Move it, along with the chunkKvStore.getChunkKey call, into a private getChunkStoreKey so the read and write paths cannot drift apart. The key type is unknown because sharded stores use structured keys.
…ead of conditional no-cache Replace the cacheMode read option and the requireRevalidatedReads flag threaded from the voxel-edit controller down to the zarr download with the approach already used for GCS: append a random query string parameter (ignored by S3) to read/stat URLs so cached responses are never used. Besides staleness after writes, this also avoids 304 revalidation responses whose Access-Control-Allow-Origin header may be stale (https://bugs.chromium.org/p/chromium/issues/detail?id=1214563#c2), which applies to S3 since its CORS headers vary with the Origin.
… arrays supportsWriting was computed by duck-typing the base kvstore before the metadata was even read, so s3 + sharded zarr reported writable even though the sharded kvstore is read-only and the first write fails. Derive it from the parsed codec chain instead: every scale must be free of sharding and of array->array codecs (which encodeArray rejects).
briossant
force-pushed
the
feature/voxel-annotation
branch
from
August 19, 2026 17:28
434477d to
88046fd
Compare
Leftover debugging guard from the [NaN,NaN,NaN] preview-position bug; the underlying cause was fixed and the per-draw check is not worth keeping. Requested in PR google#858 review.
Author
Today I don't see a possible way that a path would include Also seems like there is now too much commits for the CLA, I will squash the history. |
The fetch URL parser normalizes dot segments (even percent-encoded ones), so a key containing "." or ".." would silently address an object outside the dataset prefix. Now that the S3 driver also writes and deletes, read/stat/write/delete all reject such keys. Requested in PR google#858 review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This Draft Pull Request introduces an interactive voxel annotation feature, allowing users to perform manual segmentation by painting directly onto volumetric layers. This implementation is based on the proposal in Issue #851 and incorporates the feedback from @jbms.
Here is a live demo to try the feature, watch out, there is persistent storage, so your annotations will be saved and will override the ones already present: OPEN DEMO VIEWER
Key Changes & Architectural Overview
Following the discussion, this implementation has been significantly revised from the initial prototype:
voxlayer type, the voxel editing functionality is now integrated directly intoImageUserLayerandSegmentationUserLayervia aUserLayerWithVoxelEditingMixin. This mixin adds a new "Draw" tab in the UI.New Tool System: The Brush and Flood Fill tools are implemented as toggleable LayerTools, while the Picker tool is a one-shot tool. All integrate with Neuroglancer's new tool system. The drawing action is bound to Ctrl + Left Click.
Optimistic Preview for Compressed Chunks: To provide immediate visual feedback and solve the performance problem with compressed chunks, edits are now rendered through an optimistic preview layer.
InMemoryVolumeChunkSource.RenderLayer(e.g.,ImageRenderLayerorSegmentationRenderLayer). This ensures the preview perfectly matches the user's existing shader and display settings.Data-flow
sequenceDiagram participant User participant Tool as VoxelBrushTool participant ControllerFE as VoxelEditController (FE) participant EditSourceFE as OverlayChunkSource (FE) participant BaseSourceFE as VolumeChunkSource (FE) participant ControllerBE as VoxelEditController (BE) participant BaseSourceBE as VolumeChunkSource (BE) User->>Tool: Mouse Down/Drag Tool->>ControllerFE: paintBrushWithShape(mouse, ...) ControllerFE->>ControllerFE: Calculates affected voxels and chunks ControllerFE->>EditSourceFE: applyLocalEdits(chunkKeys, ...) activate EditSourceFE EditSourceFE->>EditSourceFE: Modifies its own in-memory chunk data note over EditSourceFE: This chunk's texture is re-uploaded to the GPU deactivate EditSourceFE ControllerFE->>ControllerBE: commitEdits(edits, ...) [RPC] activate ControllerBE ControllerBE->>ControllerBE: Debounces and batches edits ControllerBE->>BaseSourceBE: applyEdits(chunkKeys, ...) activate BaseSourceBE BaseSourceBE-->>ControllerBE: Returns VoxelChange (for undo stack) deactivate BaseSourceBE ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC] activate ControllerFE ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys) note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit. ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys) deactivate ControllerFE ControllerBE->>ControllerBE: Pushes change to Undo Stack & enqueues for downsampling deactivate ControllerBE loop Downsampling & Reload Cascade ControllerBE->>ControllerBE: downsampleStep(chunkKeys) ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC] activate ControllerFE ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys) note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit. ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys) deactivate ControllerFE end5. Dataset creation To complete Neuroglancer's writing capabilities, a dataset metadata creation/initialization feature was introduced.The workflow is triggered when a user provides a URL to a data source that does not resolve:Neuroglancer recognizes the potential intent to create a new dataset and prompts the user:Finally, the user is able to access dataset creation form:Data sources & Kvstores
Currently, there is a very limited set of supported data sources and kvstores, which are:
opfs: in-browser storage, also used for local development at some point, the relevancy can be discussed.ssa+https: a kvstore linked to an in development project, which is a stateless (thanks to OAuth 2.0) worker providing signed urls to read/write in s3 storesLimitations
Open Questions & Future Work
This PR focuses on establishing the core architecture. Several larger topics from the original discussion are noted here as future work:
Checklist
[ ] Added support to more (every?) datasources and kvstoresEdits