refactor(deps): replace RocksDB package

[rubo]

⚠️ DO NOT commit anything or merge master into this branch

Changes

Replaces the RocksDB package with our in-house Nethermind.RocksDbBindings, and moves from RocksDB v10 to v11.

New interop

• Over 1700 source-generated static P/Invoke functions over the C API under [assembly: DisableRuntimeMarshalling]: blittable signatures, no marshalling stubs, no Native.Instance singleton hop per call.
• Native objects sits behind a SafeHandle. Children (iterators, snapshots, checkpoints, WAL iterators) hold a ref-counted lease on the database handle, so a leaked iterator can no longer keep the DB open and its LOCK held.
• Native calls take a HandleLease, closing the check-then-use races the old code had
• Managed wrappers for everything we used to reach past the package for

Gone

• RocksDbExtensions.cs, and all hand-rolled rocksdb_get / rocksdb_free / pinnable-slice interop.
• All 5 unsafe blocks in DbOnTheRocks, both Marshal.StringToHGlobalAnsi/FreeHGlobal pairs, and the private FlushOptions finalizer class.
• AllowUnsafeBlocks from Nethermind.Db.Test.
• DbOnTheRocks' Native? rocksDbNative constructor parameter: source-breaking for out-of-tree subclasses; repair is now a protected virtual RepairDb.

Correctness

• Empty values no longer leak. master's GetSpan handed back a zero-length span over a live rocksdb_get allocation that no caller ever released; the bindings free it.
• Db.SharedBlockCacheSize=0 is now a config error. It used to abort() inside RocksDB.
• WarmupFile tolerates null live-file metadata; HyperClockCacheWrapper.GetUsage no longer races disposal.

Remarks

• NuGet feed change is temporary and will be reverted to production before merging.
• This is the first wave of change, more optimizations to come later.

Types of changes

What types of changes does your code introduce?

• Bugfix (a non-breaking change that fixes an issue)
• New feature (a non-breaking change that adds functionality)
• Breaking change (a change that causes existing functionality not to work as expected)
• Optimization
• Refactoring
• Documentation update
• Build-related changes
• Other: Description

Testing

Requires testing

• Yes
• No

If yes, did you write tests?

• Yes
• No

Notes on testing

Requires thorough manual testing

[claude]

Claude finished @rubo's task in 10m 27s —— View job

---

Review: chore(deps): replace RocksDB package

• Gather PR diff and context
• Review DbOnTheRocks interop → managed bindings migration
• Review reader/iterator/pinned-slice lifetime handling
• Review column-family, options and config changes
• Review tests and packaging changes
• Check CI signal

Net −244 lines with the hand-rolled Native.Instance P/Invoke, Marshal.AllocHGlobal bound buffers, the private FlushOptions finalizer class and RocksDbExtensions all gone — that is a solid direction, and a few spots are strictly better than master (RocksdbSortedView now lets the read options own the bound buffers with a correct dispose order; HyperClockCacheWrapper.GetUsage replaces a check-then-use IsClosed race with DangerousAddRef; WarmupFile handles null metadata). Findings below are mostly about hand-written safety guards that were dropped along with the interop they guarded, plus two red CI gates.

Because the bindings package isn't on a public feed yet I couldn't restore it here, so anything that depends on the new API's exact contract is phrased as "please confirm" rather than asserted.

---

🔴 Critical / blocking CI

1. nix/nuget-deps.json was not regenerated — Nix build fails on all 3 platforms. It still pins RocksDB 10.10.1.1747 (line 898) and has no Nethermind.RocksDbBindings entry. Regeneration itself then fails: error NU1101: Unable to find package Nethermind.RocksDbBindings. No packages exist with this id in source(s): _nix. PackageSourceMapping is enabled, the following source(s) were not considered: nuget.org (run 32022803943). This unblocks itself once the package is on the production feed, but the deps file still has to be regenerated in the same commit that flips the feed.

2. Code Lint fails on IDE0005 for the now-unused using System.Diagnostics; (DbOnTheRocks.cs:9) — commented inline.

🟠 High

3. GetCStyleWithColumnFamily lost the "output buffer too small" guard (DbOnTheRocks.cs:906-914). IReadOnlyKeyValueStore.Get(key, output, flags) documents "Throws if output is not large enough"; the new body is length < 0 ? 0 : length. If the bindings signal an undersized buffer with a negative code, it is silently reported as key-not-found, and the flat-state readers (BaseFlatPersistence.Reader.GetAccount / TryGetStorage) treat 0 as "no account / no slot" — a loud buffer error becomes a silently missing state read. Restore an explicit length > output.Length throw regardless of the binding contract, plus tests pinning both cases. Details · Fix this →

🟡 Medium

4. Pinned-slice leak on the "found, but null value span" path (DbOnTheRocks.cs:966-978). The old code destroyed the slice when rocksdb_pinnableslice_value returned null; now the slice is always detached, and the only consumer returns early without releasing when the span is null (KeyValueStoreRlpExtensions.cs:103-107). Either guarantee a successful TryGetPinned never yields a null span, or restore the branch. Same ownership question applies to _db.GetSpan/_db.DangerousReleaseMemory now that the repo's own rocksdb_get/rocksdb_free pair is gone. Details

5. Test coverage doesn't cover the boundaries that changed. The test diff is adaptation only (exception types, a repair-tracking subclass). Every hand-written null/empty/oversize path that this PR deleted is untested: AssertCanGetViaAllMethod only ever reads with an exactly-sized buffer. One round-trip test storing an empty value and reading it back via Get, GetSpan, the C-style Get and GetNativeSlice, plus the undersized-buffer case, would pin down findings 3 and 4 in CI instead of in production.

6. The exact-version pin was dropped — RocksDB was [10.10.1.1747], the replacement is an open range (Directory.Packages.props:69). For a native storage engine, keep the exact range so a rebuild can't silently change the on-disk format. Details

🔵 Low

7. HyperClockCacheWrapper now owns the same native handle twice (Cache + SafeHandle), and ReleaseHandle() touches the managed Cache from the finalizer thread — safe only if Cache.Dispose() is idempotent post-finalization. Holding Cache directly would remove the layer. The _nativeCacheLock removal is also a silent behaviour change. Details
8. GetWithIterator(..., IColumnFamilyHandle? _, ...) — the CF argument is dead; drop it rather than naming it _. Details
9. Nethermind.Db.Test.csproj enables AllowUnsafeBlocks for two raw-Native asserts, with no comment justifying the unsafe (required by .agents/rules/robustness.md). Details
10. A null statistics string is now silently skipped while a null compaction-stats string warns (DbMetricsUpdater.cs:37-44). Details
11. GetAllCore projections mix iterator.Value() and iterator.GetValueSpan().ToArray() (DbOnTheRocks.cs:1086 vs 1092) — pick one.
12. GetRocksDbVersion() derives the startup log line from the bindings' AssemblyVersion ("package version matches the native library version"). Please confirm 11.8.1-preview.68 ships AssemblyVersion 11.8.1.x, otherwise RocksDB: v… in the log becomes wrong.
13. DbOnTheRocks' public constructor lost the Native? rocksDbNative parameter — source-breaking for any out-of-tree subclass. Worth a release note (net surface reduction, so no objection to the change itself).

Verification checklist for the manual testing round (native v10 → v11)

These can't be checked from the diff, but they're where a bindings swap actually bites:

• Codec parity: configs require kSnappyCompression, kLZ4Compression and block_based_table_factory.format_version=5 (Config/DbConfig.cs:44,66,191,315,421) and kSnappyCompression for blob files. A native build missing Snappy/LZ4/Zstd makes existing databases unreadable at open.
• RID coverage: linux-x64/arm64, win-x64, osx-arm64 all resolve the same lockfile entry — confirm native assets exist for each.
• Round trip: open an existing v10 mainnet datadir under v11 and confirm what happens on rollback to a v10 build (new SSTs written by v11 may not be readable by the previous release) — that's release-note material either way.
• Long-running handle/memory check: several finalizer safety nets changed (FlushOptions, ReadOptions, row Cache), so a multi-hour sync watching native memory and _allocatedSpan is worth more than a short smoke run here.

---

Verdict: not mergeable yet — two red gates (Nix deps file, lint) plus the dropped buffer-size guard. The interop cleanup itself reads well.
· branch feature/rocksdb-bindings

[github-actions]

EVM Opcode Benchmark Diff

Aggregated runs: base=1, pr=1

No significant regressions or improvements detected.

[rubo]

@claude re-review. Ignore nix failures and unpinned package version, the latter is intentional.

Because the bindings package isn't on a public feed yet I couldn't restore it here

Not true. The feed is public, just not the production one. See nuget.config. The package repo is indicated in the PR description, so you can get a better picture.

The following findings are rejected:

• 3 (High, "buffer-too-small guard dropped") is not a real defect. RocksDb.Get(key, destination, cf, readOptions) (src/Nethermind.RocksDbBindings/RocksDb.cs:465) implements exactly the guard that was deleted — it throws ArgumentException when the value doesn't fit and returns -1 only for a missing key. length < 0 ? 0 : length is therefore the correct adaptation and IReadOnlyKeyValueStore.Get's documented contract holds. The new test pins both branches.

• 4 (Medium, "pinned-slice leak on null value span") is also handled inside the bindings: TryGetPinned (RocksDb.cs:379-384) destroys the slice and returns false when rocksdb_pinnableslice_value yields null, so GetNativeSlice can never hand out a detached handle with a null span. GetSpan likewise frees zero-length allocations itself. The empty-value test asserts _allocatedSpan == 0 after the round trip.

• 12 (RocksDB version log) checks out — the package's AssemblyVersion is 11.8.1.0, so Program.cs:232 logs RocksDB: v11.8.1.

[claude]

Claude finished @rubo's task in 7m 41s —— View job

---

Re-review: chore(deps): replace RocksDB package

• Fetch PR diff and the follow-up commit (584fc8e)
• Pull NethermindEth/rocksdb-bindings source and verify the rejected findings
• Re-check the open findings
• Look for anything new
• Post verdict

You were right that I could read the bindings — I pulled RocksDb.cs, PinnedSlice.cs, Cache.cs, ReadOptions.cs, MergeOperator.cs and ColumnFamilyOptions.cs from the repo and checked each rejection against the source.

---

Rejections — all three confirmed correct

3 (buffer-too-small guard). Verified, my finding was wrong. RocksDb.Get(key, destination, cf, readOptions) (RocksDb.cs:465-483) is TryGetPinned → return -1 on miss → throw new ArgumentException when value.Length > destination.Length → CopyTo in a try/finally that disposes the slice. That's the deleted guard, moved one layer down and documented on <exception>. length < 0 ? 0 : length is the right adaptation and IReadOnlyKeyValueStore.Get's contract holds. Get_into_output_buffer_reports_missing_key_and_rejects_undersized_buffer pins both branches.

4 (pinned-slice leak). Also verified. TryGetPinned (RocksDb.cs:379-384) destroys the slice and returns false when rocksdb_pinnableslice_value yields null, so GetNativeSlice can never detach a handle over a null span — my "both branches are bad" framing didn't hold. GetSpan (RocksDb.cs:437-442) frees the zero-length allocation itself, and DangerousReleaseMemory no-ops on empty, so the _allocatedSpan/GC.AddMemoryPressure accounting in GetSpanWithColumnFamily stays balanced. The only GetNativeSlice consumer in the tree (KeyValueStoreRlpExtensions.cs:103) gates on IsNull(), not IsEmpty, so a stored empty value still reaches the finally that releases. Can_read_back_empty_value covers it.

12 (version log). Accepted on your word; nothing in the diff contradicts it.

Also confirmed while I was in there: the Unsafe.As<byte, byte> hack in MergeOperatorAdapter is genuinely obsolete — IMergeOperator declares out byte success and the native shim writes *success = succeeded through a byte*, so the old 1-vs-4-byte bool clobber is gone. Merge operators are rooted by a real GCHandle in the state struct, so no lifetime regression there either. And ReadOptions.SetIterateBounds copies into NativeMemory it owns and frees in ReleaseHandle, which is a faithful replacement for the deleted Marshal.AllocHGlobal pair.

Previous findings — status

#	Status
1 Nix deps	excluded per your instruction
2 Code Lint IDE0005	✅ fixed — Code Lint green
3 buffer guard	✅ withdrawn
4 pinned slice	✅ withdrawn
5 test coverage	✅ addressed — two new tests cover the empty-value round trip and the undersized buffer
6 version pin	intentional, dropped
7 HyperClockCacheWrapper	mostly resolved; see the new finding below
8 dead cf parameter	✅ dropped
9 AllowUnsafeBlocks	✅ justification comment added
10 asymmetric stats logging	✅ else-warn added (nit inline)
11 mixed iterator projections	✅ both use GetValueSpan().ToArray()
12 version log	✅ confirmed
13 ctor surface change	still a release-note item

New findings

🟠 Medium — HyperClockCacheWrapper now throws on a zero capacity. The rewrite dropped the if (_capacity > 0) guard around GC.AddMemoryPressure, which throws ArgumentOutOfRangeException for bytesAllocated <= 0. Db.SharedBlockCacheSize (DbModule.cs:73) and FlatDb.BlockCacheSizeBudget (FlatRocksDbConfigAdjuster.cs:46,55) are both operator-settable ulongs, so 0 — the obvious way to disable the shared cache — goes from "works, pressure accounting skipped" on master to a startup crash out of the DI factory. No test covers capacity 0, so CI won't catch it. Details + suggestion · Fix this →

Same comment covers two smaller points: _disposed is a plain bool, so concurrent Dispose() can double-call GC.RemoveMemoryPressure (SafeHandle used to guarantee exactly-once); and there's no finalizer, so an undisposed wrapper never removes its pressure — harmless today since both call sites dispose, and the native handle is still covered by CacheHandle's critical finalizer.

🔵 Low — the new logger.Warn at DbMetricsUpdater.cs:46 is missing the if (logger.IsWarn) guard used elsewhere in this layer, and it fires every StatsDumpPeriodSec per updater (one per column family). Details

🔵 Low — src/Nethermind/Nethermind.Runner/packages.lock.json lost its trailing newline (\ No newline at end of file at the end of the diff). Regeneration artifact, worth restoring so the next dotnet restore --force-evaluate doesn't churn the line back.

🔵 Low — RepairDb is a new protected virtual reached from the constructor path (Init → RepairIfCorrupted), and RepairTrackingDbOnTheRocks overrides it using a primary-constructor parameter. That works only because Roslyn assigns primary-ctor captures before the base constructor call; a hand-written derived constructor assigning a field the normal way would observe null here. The existing // ReSharper disable once VirtualMemberCallInConstructor already flags the pattern, so this is a note rather than a request — it's a reasonable test seam and net less surface than the old Native injection.

Observation, not a finding — BuildOptions runs once per column family (DbOnTheRocks.cs:172), so with RowCacheSize > 0 on a ColumnsDb each call overwrites _rowCache and only the last Cache is disposed in ReleaseUnmanagedResources. On master this was an outright IntPtr leak; now the orphans are reclaimed by CacheHandle's critical finalizer, so the PR strictly improves it. Mentioning only because you're in the file.

CI

Code Lint, Code formatting, Build solutions, Integration tests (E2E), Stateless execution tests, Trivy, spell check, dependency review all ✅. Nix ❌ as expected (excluded). Flat DB tests, extra test variants, EF tests and the Hoodi sync gate were still running when I looked.

Manual-testing checklist (native v10 → v11)

Still the part the diff can't answer, unchanged from last round: codec parity (kSnappyCompression / kLZ4Compression / format_version=5 in Config/DbConfig.cs), native assets for all five RIDs in the lockfile, opening an existing v10 datadir under v11 and what a rollback to a v10 build does with v11-written SSTs, and a multi-hour sync watching native memory plus _allocatedSpan now that the FlushOptions / ReadOptions / row-Cache finalizer safety nets all changed shape.

---

Verdict: the migration itself holds up — I was wrong on both of the substantive findings, and the follow-up commit closed the rest. One Medium (zero-capacity crash) and three Lows remain; none is deep, and the Medium is a one-line guard.
· branch feature/rocksdb-bindings

[rubo]

@claude re-review, ignoring nix and unpinned package.

Some context:

A zero capacity makes rocksdb request a zero-length anonymous mapping for the
HyperClockCache handle table and abort the process, so Db.SharedBlockCacheSize=0
took the node down with a bare native message and exit code 0xC0000409. Validate
it in HyperClockCacheWrapper instead, where it can still be reported as the
configuration error it is.

Also make Dispose exactly-once via Interlocked, which SafeHandle used to
guarantee before that layer was dropped, and guard every log call in
DbMetricsUpdater with its IsWarn/IsError check.

[claude]

Re-review: chore(deps): replace RocksDB package

• Fetch the new commits since 584fc8e
• Verify the zero-capacity validation in HyperClockCacheWrapper
• Verify exactly-once Dispose via Interlocked
• Verify IsWarn/IsError guards in DbMetricsUpdater
• Re-check remaining open findings + look for anything new
• Check CI signal and post verdict

Reviewed b1fcaed (fix(db): reject a zero block cache capacity) and f6cdf01 (master merge — touches nothing in the DB layer). Nix and the unpinned package version are excluded per your instruction. All three points from your message check out.

---

The zero-capacity fix

Your framing is better than mine — I described master as "works, pressure accounting skipped", but if rocksdb abort()s on the zero-length handle-table mapping then master's behaviour was a bare native message and 0xC0000409, which is strictly worse than an ArgumentOutOfRangeException. Validating in the wrapper is the right placement: it's the single choke point for both config paths (DbModule.cs:73 and FlatRocksDbConfigAdjuster.cs:48,57), and InvalidConfigurationException + ExitCodes.ForbiddenOptionValue matches the established pattern (ZkGasSchedule.cs:56,94,110). The comment carrying the native error string is exactly the kind of why that survives a git blame.

Verified, all three:

Point	Status
Zero capacity rejected as a configuration error	✅ HyperClockCacheWrapper.cs:29-34, <exception>-documented, covered by SharedCacheRejectsZeroCapacity
Dispose exactly-once via Interlocked	✅ Interlocked.Exchange(ref _disposed, 1), bool → int; SharedCacheCanBeCreatedAndDisposed double-disposes
Every DbMetricsUpdater log call guarded	✅ all four Warn + the Error now guarded (lines 46, 52, 102, 163, 226) — one unguarded Info remains, see below

Also confirmed while in there: Nethermind.Config (ExitCodes) and Nethermind.Db (IFlatDbConfig) both resolve for Nethermind.Db.Rocks, and Code Lint is green, so no repeat of the IDE0005 regression; packages.lock.json's trailing newline is restored; and DestroyReadOptions / RocksDbSharp are gone from the tree entirely (only a stale comment mentions the old package now).

Previous findings — status

#	Status
1 Nix deps	excluded per your instruction
2–5, 8–12	✅ closed in earlier rounds (3 and 4 were my errors)
6 version pin	intentional, excluded
7 zero-capacity crash (last round's Medium)	✅ fixed
_disposed race	✅ fixed
unguarded Warn (last round's Low)	✅ fixed
packages.lock.json newline	✅ fixed
13 ctor surface change	still a release-note item (DbOnTheRocks' public ctor lost Native? rocksDbNative)
RepairDb virtual-from-ctor	informational, unchanged

New findings — all Low

🔵 Three comments in ColumnsDb.cs went stale with this change. :167-168 still says "ReadOptions in RocksDbSharp has a finalizer but no IDisposable" — names the removed package, and the claim is now false (the bindings' ReadOptions is a SafeHandle, which is why line 260 can Dispose() it). :174-175 says each GetViewBetween ReadOptions is finalizer-only and "not addressed by this PR" — RocksdbSortedView.Dispose() now disposes it, and per AGENTS.md comments shouldn't reference the PR. Details

🔵 The guard rejects exactly zero — is zero the whole unsafe range? Cache.CreateHyperClock passes estimatedEntryCharge = 0, so rocksdb auto-sizes the handle table from capacity alone. FlatRocksDbConfigAdjuster.cs:46 computes BlockCacheSizeBudget * 0.3, so a small budget yields a handful of bytes; if the auto-sizing rounds down, a tiny-but-nonzero capacity hits the same mapping. The new test only proves 10 KiB clears the floor. Same comment also covers: the message renders bare key names (SharedBlockCacheSize) rather than the Db. / FlatDb. prefixes operators actually type, and it's worth one manual --Db.SharedBlockCacheSize 0 run to confirm exit code 107 actually reaches the process — the throw is wrapped in Autofac's DependencyResolutionException, and only EthereumStepsManager.TryUnwrapException unwraps that, on the step-construction path only. Details

🔵 logger.Info at DbMetricsUpdater.cs:213 is the one call left unguarded, and the priciest — LogMemoryProfile() runs ~7 + numLevels native GetProperty calls per tick before formatting. An early if (!logger.IsInfo) return; at line 178 skips the queries, not just the string. Pre-existing on master. Details

🔵 Test coverage stops at the wrapper. SharedCacheRejectsZeroCapacity pins the constructor, but not the operator-facing wiring. A case in the existing FlatRocksDbConfigAdjusterTests with BlockCacheSizeBudget = 0 would pin that the config path reports it as a configuration error rather than something Autofac-shaped.

CI

Build solutions, Code Lint, Code formatting, Stateless execution tests, Trivy, spell check, dependency review, Build tools, PR labeler — all ✅. Nix ❌ as expected (excluded). I couldn't build or run tests locally (dotnet build needs approval in this environment), so the compile/test signal here is CI's.

Manual-testing checklist (native v10 → v11)

Unchanged, and still the part the diff can't answer: codec parity (kSnappyCompression / kLZ4Compression / format_version=5 in Config/DbConfig.cs), native assets for all five RIDs in the lockfile, opening an existing v10 datadir under v11 and what a rollback to a v10 build does with v11-written SSTs, and a multi-hour sync watching native memory plus _allocatedSpan. Add to that: one deliberate zero-capacity startup to confirm the new error path prints cleanly and exits 107.

---

Verdict: mergeable from my side once Nix is regenerated on the production feed. The Medium from last round is fixed, the two smaller points with it, and nothing new above Low turned up — the four remaining items are comment hygiene, a floor question for the new guard, one pre-existing log guard, and an optional test.

View job run · branch feature/rocksdb-bindings

[rubo]

@claude review ignoring nix and unpinned package

[claude]

Claude finished @rubo's task in 9m 13s —— View job

---

Re-review: chore(deps): replace RocksDB package

• Fetch the new commits since b1fcaed
• Review the write-options getters refactor (f4122b0)
• Review the stale-comment / config-path commit (66fea0c)
• Check the bindings update (cc14ece)
• Re-check remaining open findings + look for anything new
• Check CI and post verdict

Reviewed f4122b0, cc14ece and 66fea0c on top of the previous round. Nix and the unpinned package version are excluded per your instruction. Nothing new above Low, and one of my previous Lows was simply wrong — details below.

---

The three new commits

f4122b0 — managed write-option getters. Better than the comment I asked for: AllowUnsafeBlocks is gone from Nethermind.Db.Test.csproj entirely, so there is no unsafe block left to justify and no test coupling to Nethermind.RocksDbBindings.Native. SetDisableWal(true) / SetLowPriority(true) are faithful replacements for DisableWal(1) / rocksdb_writeoptions_set_low_pri(…, 1).

cc14ece — bump to preview.71. Not cosmetic: preview.68 had no SetDisableWal/GetDisableWal/GetLowPriority, which is exactly why every build job on f4122b0 was red (error CS1061: 'WriteOptions' does not contain a definition for 'SetDisableWal', run 32047326090). Green again on 66fea0c.

66fea0c — stale comments + config-path coverage. The three ColumnsDb comments now read correctly against master (no package name, no "this PR", and the GetViewBetween note matches RocksdbSortedView.Dispose()), and the error message carries the Db. / FlatDb. prefixes an operator actually types.

FlatDatabase_WithUnusableCacheBudget_ReportsConfigurationError also closes my "is the floor exactly zero?" question. The budget = 2 case is the interesting one: 2 * 0.3 → 0, so it pins that a nonzero but tiny budget truncating to zero is still reported as a configuration error rather than aborting. And since CreateHyperClock leaves estimatedEntryCharge = 0, the auto-sized handle table's length scales with capacity, so any capacity ≥ 1 yields at least one slot and a nonzero mapping — capacity == 0 really is the whole unsafe range. Considered resolved.

Correction to my last round

The logger.Info "unguarded call" (last round's third Low) was wrong. LogMemoryProfile() already starts with if (!logger.IsInfo) return; — on master as well as here (DbMetricsUpdater.cs:180) — so the GetProperty calls were never running with Info off. Your "It's okay as-is" was correct; withdrawn.

Verified while re-reading

• batch.DataSize is rocksdb_writebatch_data's size out-param (WriteBatch.cs:51-59), not the entry count — so the 16 KiB write-batch reuse threshold keeps identical semantics to the interop it replaced. This was the one silent-behaviour-change candidate left in the diff.
• GetSpanWithColumnFamily is now strictly better than master on empty values: master's RocksDbExtensions.GetSpan returned a zero-length span over a real rocksdb_get allocation that no caller released (!span.IsNullOrEmpty() skipped the accounting, and consumers skip DangerousReleaseMemory on empty) — a small per-read leak. RocksDb.GetSpan frees the zero-length allocation itself, which Can_read_back_empty_value's AllocatedSpan == 0 assertion now pins.
• PinnedSlice.DangerousDestroy ignores 0, so dropping master's if (handle != default) from DangerousReleaseHandle is safe.
• No RocksDbSharp / Native.Instance / rocksdb_* call site left anywhere in src/Nethermind.
• FetchTotalPropertyValue's switch from string-parse to TryGetIntProperty is safe for all six property names used (all are int properties in DB::Properties); the only behaviour delta is that an unavailable property now yields 0 silently instead of surfacing as a caught-and-warned exception.

Previous findings — status

#	Status
1 Nix deps	excluded per your instruction
2–5, 8, 10–12	✅ closed in earlier rounds (3 and 4 were my errors)
6 version pin	intentional, excluded
7 zero-capacity crash + _disposed race	✅ fixed in b1fcaed
9 AllowUnsafeBlocks	✅ better than asked — the unsafe is gone
stale ColumnsDb comments	✅ fixed
zero-capacity floor question	✅ answered by the budget = 2 test case
unguarded Info	❌ my error, withdrawn
packages.lock.json newline	⚠️ regressed in cc14ece, see below
13 ctor surface change	still a release-note item (DbOnTheRocks' public ctor lost Native? rocksDbNative)
RepairDb virtual-from-ctor	informational, unchanged

New findings — both Low

🔵 packages.lock.json lost its trailing newline again. cc14ece re-introduced \ No newline at end of file; it had been restored last round. Rides along with the regeneration that flips the feed. Details

🔵 Eight per-DB native option handles are still finalizer-only. ReleaseUnmanagedResources now disposes _db, _iteratorManager, _rowCache and _reader, but WriteOptions, _noWalWrite, _lowPriorityWriteOptions, _lowPriorityAndNoWalWrite, _defaultReadOptions, _hintCacheMissOptions, _readAheadReadOptions and DbOptions are left to their critical finalizers. Identical to master in effect (no leak — SafeHandle reclaims them), so this is purely a consistency note: they're all IDisposable now, and disposing them after _db.Dispose() would finish the deterministic-cleanup theme the rest of the PR follows. Entirely optional.

CI (all on 66fea0c)

Build solutions, Build tools, Code Lint, Code formatting, Integration tests (E2E), Nethermind tests (Flat DB), Nethermind extra test variants, Nethermind/Ethereum tests, Stateless execution tests, EVM Opcode Benchmark Diff, Sync PR Gate (Hoodi), spell check, dependency review, PR labeler — all ✅.

Two ❌:

• Nix — excluded per your instruction.
• Trivy scanner — the Scan step succeeded; only Upload scan results failed, with ##[error]No server is currently available to service your request (GitHub code-scanning outage at 17:47). Transient infra, not this PR — a rerun should clear it.

Worth noting the EXPB benchmark signal is still missing: the "No metrics were produced" comment and the failed Run EXPB Reproducible Benchmarks / Publish Docker image runs were all triggered on f4122b0, i.e. the commit that didn't compile. Since the PR carries a performance is good label and lists Optimization as a change type, re-triggering EXPB on 66fea0c is the cheapest way to get an actual number.

Manual-testing checklist (native v10 → v11)

Unchanged and still the part the diff can't answer: codec parity (kSnappyCompression / kLZ4Compression / format_version=5 in Config/DbConfig.cs), native assets for all five RIDs in the lockfile, opening an existing v10 datadir under v11 and what a rollback to a v10 build does with v11-written SSTs, a multi-hour sync watching native memory plus _allocatedSpan, and one deliberate --Db.SharedBlockCacheSize 0 startup to confirm the new error prints cleanly and exits 107 (the throw is still wrapped in Autofac's DependencyResolutionException, and the new adjuster test calls GetForDatabase directly, so it doesn't cover the container path).

---

Verdict: mergeable from my side once Nix is regenerated on the production feed. Everything from the previous rounds is closed, one of my own Lows was wrong, and the two remaining items are a missing newline and an optional consistency cleanup.
· branch feature/rocksdb-bindings

[rubo]

@claude review, ignoring nix, unpinned version, and new line in packages.lock.json (it's autogenerated)

On your last finding

Eight per-DB native option handles are still finalizer-only.

Doing it would be a correctness risk for no benefit. Rejected.

Forward- and backward-compatibility between v10 and v11 has been tested successfully.

[claude]

Claude finished @rubo's task in 3m 42s —— View job

---

Re-review: refactor(deps): replace RocksDB package

• Fetch the new commits since 66fea0c
• Check whether the master merge disturbed anything in the DB layer
• Re-examine the full DB-layer diff with fresh eyes
• Weigh the rejected finding
• Check CI and post verdict

Nix, the unpinned package version and the packages.lock.json newline are excluded per your instruction. No new findings, and nothing open above the release-note item.

The only commit since the last round is the master merge 649da20. git diff 66fea0c..HEAD over Nethermind.Db.Rocks, Nethermind.Db.Test, Nethermind.Runner.Test, Directory.Packages.props, nuget.config and the lockfile is empty — the merge brought in BlockTree.Initializer, chain specs and a witness-capture test fix, none of which touch this layer, and there is no RocksDbSharp / Native.Instance / rocksdb_* call site anywhere in src/Nethermind after it. So the reviewed surface is unchanged, and CI below is a clean signal on the merged tree.

---

On the rejected finding — agreed, and for a stronger reason than "no benefit"

I'd looked at the eight option handles as eight independent fields. They aren't: _readAheadReadOptions is handed to the DB's own IteratorManager (DbOnTheRocks.cs:121) and to every ColumnDb's IteratorManager (ColumnDb.cs:33), and _defaultReadOptions / _hintCacheMissOptions are the options every column reader's GetSpan / GetNativeSlice / KeyExists path runs through. RocksDbReader deliberately only disposes options it created itself (_ownsReadOptions, RocksDbReader.cs:46,52), and there's no ordering guarantee between a ColumnDb and the parent DbOnTheRocks that owns those handles. Disposing them in ReleaseUnmanagedResources would be freeing native options that pooled iterators and column readers may still reference — a use-after-free for a cleanup the SafeHandle critical finalizers already perform correctly. Your call is right; withdrawn, and it shouldn't come back in a later round.

v10 ↔ v11 forward/backward compatibility tested — that closes the largest item on the manual checklist (the "what happens on rollback to a v10 build" question, which the diff genuinely could not answer). What remains from that list is only the long-running one: a multi-hour sync watching native memory and _allocatedSpan, since the FlushOptions / ReadOptions / row-Cache lifetimes all changed shape. Codec parity and RID coverage are implicitly answered by a successful v10-datadir open plus green CI on all platforms.

Fresh pass — what I re-checked and found clean

• ReleaseUnmanagedResources ordering (DbOnTheRocks.cs:1439-1452): batches → _iteratorManager → _db → _rowCache. The row cache outliving the DB it's attached to is the correct order, and Cache disposal is now deterministic where master leaked the IntPtr outright.
• _defaultFlushOptions as a static field is a faithful replacement for master's FlushOptions.DefaultFlushOptions singleton, minus the hand-written finalizer class.
• WarmupFile (:233-238) handles a null GetLiveFilesMetadata() with a warn-and-skip instead of an NRE on startup.
• FetchTotalPropertyValue in both DbOnTheRocks and ColumnsDb moved from long.TryParse(GetProperty(...)) to TryGetIntProperty — same 0 fallback, one less string allocation per metric tick per CF.
• MergeOperatorAdapter.GetResult: the data is null path sets success = 0 and resultLength = 0 before returning nint.Zero, so the native shim can't read an uninitialised length; the success path allocates with NativeMemory.Alloc and frees through DeleteValue. Balanced.
• ColumnDb.SetWriteBuffer / ColumnsDb.ApplyOptions now go through _db.SetOptions(cf, …), which surfaces a native error instead of the raw rocksdb_set_options_cf whose out-error was on the caller.
• Can_read_back_empty_value exercises Get, the C-style Get, GetSpan and GetNativeSlice on a stored empty value and asserts AllocatedSpan == 0 — that's the assertion that pins the two findings I got wrong in round one.

Open items

Item	Status
Nix deps regeneration	excluded (rides along with the production-feed flip)
Version pin	intentional
packages.lock.json newline	autogenerated, ignore
Eight finalizer-only option handles	✅ withdrawn — rejection is correct
DbOnTheRocks public ctor lost Native? rocksDbNative	release-note item for out-of-tree subclasses (net surface reduction, no objection)
RepairDb virtual-from-ctor	informational

CI — all on the merge commit 649da20

Build solutions, Build tools, Code Lint, Code formatting, Integration tests (E2E), Nethermind tests (Flat DB), Nethermind extra test variants, Nethermind/Ethereum tests, Stateless execution tests, Trivy, spell check, dependency review, PR labeler — ✅. Nix ❌ (excluded). EVM Opcode Benchmark Diff and Sync PR Gate (Hoodi) still running.

Two notes on the perf signal, since the PR carries performance is good:

• The earlier MULMOD +360% comment is noise, not a regression — the bot itself listed MULMOD under "Noisy rerun opcodes", PR CV is 19% against a 1.1% base and the uncertainty band (±53%) swamps the ±5% threshold. A pure-EVM opcode has no path through this diff.
• EXPB still has no numbers. The "No metrics were produced" comment came from a run triggered on f4122b0, the commit that didn't compile (WriteOptions lacked SetDisableWal before preview.71). Re-triggering on 649da20 is the cheapest way to get a real figure for an Optimization-labelled PR.

---

Verdict: mergeable from my side once Nix is regenerated on the production feed. Nothing new this round, the last open finding was correctly rejected, and the v10↔v11 compat result retires the riskiest untestable item.
· branch feature/rocksdb-bindings

[kamilchodola]

Benchmark summary — EXPB + RPC (eth_call corpus, EthCallChaos, multicall)

Tested: 6b3d571208 (image rocksdb-bindings-6b3d571) vs its own merge-base master-2c15b67, flat layout, all runs paired and order-balanced (A,B / B,A / A,B) so the arm-position effect can't masquerade as a delta. Both arms pinned to immutable image tags.

Head has since moved to 649da20d73 (package preview.68 → preview.71 + 3 more DB commits). The linux-x64 native in preview.71 is byte-identical to the one tested, so the engine/allocator findings carry forward; the managed-side changes since (read write options through the managed getters, reject a zero block cache capacity) are not covered by these numbers.

This is three changes at once

Binding layer + native engine 10.10.1 → 11.8.1 + RocksDB's allocator glibc → jemalloc. The numbers say "this is faster", not "jemalloc bought us X".

jemalloc is confirmed live, statically linked and internalized (0 of 1749 exported dynsyms are allocator entry points → serves RocksDB only, does not interpose the process allocator). Verified via RocksDB itself: with --Db.AdditionalRocksDbOptions="dump_malloc_stats=true;" the branch emits ___ Begin jemalloc statistics ___ every stats dump, master emits nothing (DumpMallocStats compiles to a no-op without ROCKSDB_JEMALLOC). Note a je_* symbol scan proves nothing here — the native is stripped.

Correctness — clean

check	result
private 497-record eth_call corpus parity	497/497, 0 content mismatches, over 10 runs, both directions (master-as-baseline and branch-as-baseline)
EXPB benchmark jobs	27 jobs, 0 exceptions, 0 invalid blocks
live mainnet (DevNode-rubo-r3243 / r4288)	synced, matching heads and hashes, 0 bad blocks, banner RocksDB: v11.8.1; 12 h scan for corrupt / ioerror / bad block / state-root-mismatch → nothing

EXPB (n=3 paired, flat)

suite	AVG	MED	P90	P99	pairs favouring branch
fusaka 1k	−2.96% (p=0.030)	−2.47%	−2.22% (p=0.034)	−7.41% (p=0.033)	3/3
realblocks 1k	−5.01% (p=0.079)	−3.96%	−6.67% (p=0.045)	−5.19%	3/3
superblocks	−10.81% (p=0.021)	−9.29% (p=0.013)	−11.23% (p=0.031)	3/3

9/9 pairs favour the branch on AVG (sign test p=0.004), in both arm orders. An earlier campaign on the previous baseline gave −2.2 / −4.4 / −9.2% AVG — reproduced, marginally larger.

eth_call

condition	result
warm (after 240 s warmup)	~0% — noise, flips sign with arm order
cold, no warmup (n=3, and n=4 in the earlier campaign)	−1.7% p50 (3/3 and 4/4 favour branch; avg p=0.33 → directional, not significant)
saturated capacity, 250/500 rps	+2–6% throughput

A warm-only eth_call A/B reports "neutral" here — the effect lives in the cold path. Disable the warmup cell to see it.

multicall (order-balanced, both arm orders) — updated with the re-run on the current baseline

rps	pre-refactor image (56736ef6)	current baseline (6b3d571)
100	noisy, mixed	−0.4% (4/6 scenarios >2 ms favour branch)
250	−8.0% (7/7)	−2.0% (6/7)
500	−2.5% (7/7)	−0.5% (4/7)

Per-scenario on the current baseline @250 rps: high-gas#1/#2/#3 −1.3 / −2.8 / −1.2%, large#1 +0.1%, med#1 −3.3%, small#1 −1.3%, small#2 −4.1%.

The earlier −8% at 250 rps came from the pre-refactor image; on the current baseline the multicall effect is neutral-to-slightly-better, not a win. Both arms moved (master @250 high-gas#1 1626.9 → 1586.9 ms, branch 1540.4 → 1565.3), and per-order spreads are wide today (e.g. high-gas#1 @100: +5.8% / −9.2%), so position noise dominates an effect of this size. Scenarios under ~2 ms (multicall/small#2) are excluded from the means — their relative deltas are sub-millisecond jitter.

Shape of the win

Monotonic in how much DB work the request does: warm eth_call ~0 → cold eth_call −2% → fusaka −3% → realblocks −5% → saturated multicall −8% → superblocks −11%. Consistent with the allocator/engine acting on the RocksDB read + compaction path rather than on EVM execution.

Worth resolving before merge

1. RSS soak. MallocHelper's mallopt/malloc_trim are glibc-only, so they no longer reach the DB heap — RocksDB memory now decays on jemalloc's schedule (dirty_decay_ms/muzzy_decay_ms). Nothing here measures that; a multi-hour node run watching RSS would.
2. jemalloc version is unidentifiable — it reports 0.0.0-0-g000000missing_version_try_git_fetch_tags (built from an untagged checkout), so you can't tell 5.2 from 5.3, which differ in decay/background_thread behaviour. Worth fixing in the bindings build.
3. Nothing pins jemalloc's presence. It's an unversioned trait of a -preview package; rocksdb_jemalloc_nodump_allocator_create returns NotSupported in non-jemalloc builds, so a one-line assertion in Nethermind.Db.Test would turn this into a CI gate.
4. JE_MALLOC_CONF / MALLOC_CONF with a deliberately invalid pair produced no <jemalloc>: warning despite jemalloc running — if runtime allocator tuning is ever wanted, that path needs checking.