Skip to content

fix: prevent concurrent access to non-thread-safe maps in SketchBufferAggregatorHelper - #20090

Open
zhang-arvin wants to merge 1 commit into
apache:masterfrom
zhang-arvin:fix/issue-18040-duplicate-query-exception
Open

fix: prevent concurrent access to non-thread-safe maps in SketchBufferAggregatorHelper#20090
zhang-arvin wants to merge 1 commit into
apache:masterfrom
zhang-arvin:fix/issue-18040-duplicate-query-exception

Conversation

@zhang-arvin

Copy link
Copy Markdown

Description

Fixes #18040 - Runtime Exception when executing a query twice in a short duration.

Root Cause

The ParallelCombiner and StreamingMergeSortedGrouper are designed for concurrent read/write from different threads. However, SketchBufferAggregatorHelper used IdentityHashMap and Int2ObjectOpenHashMap, which are not thread-safe.

Concurrent access to these maps by:

  • The writing thread (via aggregate()getOrCreateUnion())
  • The reading thread (via get())

could corrupt the internal map structure, causing ArrayIndexOutOfBoundsException during sketches aggregation (e.g., Index 180 out of bounds for length 129).

Fix

Replaced IdentityHashMap with ConcurrentHashMap and Int2ObjectOpenHashMap with ConcurrentHashMap to ensure thread safety when used with ParallelCombiner.

Changes

  • extensions-core/datasketches/.../SketchBufferAggregatorHelper.java: Replace non-thread-safe maps with ConcurrentHashMap

Key Features/Changes

  • Thread-safe access to Union cache and memory cache in SketchBufferAggregatorHelper
  • Uses ConcurrentHashMap.computeIfAbsent for atomic map initialization

Verification

The existing tests for ParallelCombiner and StreamingMergeSortedGrouper cover the concurrent read/write pattern. The fix ensures that Datasketches-based aggregators work correctly under parallel combining.

…rAggregatorHelper

The ParallelCombiner and StreamingMergeSortedGrouper are designed for
concurrent read/write from different threads. However,
SketchBufferAggregatorHelper used IdentityHashMap and
Int2ObjectOpenHashMap, which are not thread-safe. Concurrent access to
these maps by the writing thread (via aggregate()) and the reading
thread (via get()) could corrupt the internal map structure, causing
ArrayIndexOutOfBoundsException.

This fix replaces IdentityHashMap with ConcurrentHashMap and
Int2ObjectOpenHashMap with ConcurrentHashMap to ensure thread safety
when used with ParallelCombiner.

Fixes apache#18040

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 1
P2 0
P3 0
Total 1

Reviewed 1 of 1 changed files.


This is an automated review by Codex GPT-5.6-Luna(max)

private final int maxIntermediateSize;
private final IdentityHashMap<ByteBuffer, Int2ObjectMap<Union>> unions = new IdentityHashMap<>();
private final IdentityHashMap<ByteBuffer, WritableMemory> memCache = new IdentityHashMap<>();
private final ConcurrentHashMap<ByteBuffer, ConcurrentHashMap<Integer, Union>> unions = new ConcurrentHashMap<>();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Do not key caches by mutable ByteBuffers

ConcurrentHashMap uses ByteBuffer.equals/hashCode, which depend on the buffer's remaining contents. Union updates mutate those contents in place, changing the key hash after insertion. Subsequent unions.get(buf) calls can miss and get() returns SketchHolder.EMPTY; distinct equal-content buffers can also alias. Preserve identity/stable-key semantics while making access concurrent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runtime Exception when executing a query twice in a short duration

2 participants