Skip to content

Experiment: Enable bloom filters for PebbleDB - #3949

Draft
brbrr wants to merge 2 commits into
mainfrom
try/db-bloom
Draft

Experiment: Enable bloom filters for PebbleDB#3949
brbrr wants to merge 2 commits into
mainfrom
try/db-bloom

Conversation

@brbrr

@brbrr brbrr commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

This is an experiment that enables bloom filters for pebble SSTables. The tricky part of this is that to populate the blooms on SSTables, the DB table needs to be written to. To effectively add blooms, we need to run a compaction across all the tables. It's quite a long-running process (>6h).

  1. Enables bloom filters everywhere — a new pebblev2.WithBloomFilter() option sets a 10-bits-per-key filter policy (~1% false-positive rate) on all LSM levels. It is applied both in node.New (the running node) and in the juno db commands' openDB. Filters only get built as sstables are written, so existing data doesn't benefit until it is compacted.
  2. Adds a juno db compact command — compacts the full key range so current table options (bloom filters, compression) apply to old data. --compression is required, --compaction-concurrency defaults to all cores, and --force handles the hard case: an already fully-compacted database, where Pebble's manual compaction would pick nothing.

Here are preliminary results of the bench run (comparing dbs with and without blooms populated):

method dbbloom p50 dbbloom p90 dbbloom p99 main p50 main p90 main p99 p50 ratio (main/dbbloom) faster
getEvents 43.19 53.17 70.22 82.69 95.83 110.95 1.91× dbbloom
getClass 17.20 37.49 239.16 19.71 39.12 238.30 1.15× dbbloom
getStorageProof 13.50 19.98 32.00 14.64 21.48 34.98 1.08× dbbloom
getCompiledCasm 11.39 46.72 234.26 11.90 49.66 243.75 1.04× dbbloom
getClassAt 21.82 39.51 237.98 22.41 39.93 242.14 1.03× dbbloom
getNonce 4.31 5.67 8.06 4.41 5.92 7.92 1.02× dbbloom
getClassHashAt 3.31 4.58 6.09 3.34 4.58 5.92 1.01× dbbloom
getTransactionStatus 5.88 8.27 11.33 5.91 8.28 11.65 1.01× dbbloom
getTransactionReceipt 6.81 9.45 12.86 6.84 9.19 13.17 1.00× dbbloom
getTransactionByHash 5.83 8.21 11.21 5.76 7.91 11.88 0.99× main
getBlockTransactionCount 3.74 4.64 6.45 3.66 4.89 7.55 0.98× main
getStorageAt 5.56 7.46 13.94 5.36 7.38 13.18 0.97× main
getBlockWithReceipts 8.85 19.65 73.19 8.51 18.62 67.98 0.96× main
getTransactionByBlockIdAndIndex 4.25 5.84 11.61 4.02 5.20 8.64 0.95× main
getStateUpdate 4.57 6.47 9.58 4.30 6.20 10.92 0.94× main
getBlockWithTxHashes 7.88 11.28 30.00 7.38 10.91 20.44 0.94× main
getBlockWithTxs 8.40 13.28 42.92 7.81 12.49 28.96 0.93× main

@brbrr brbrr self-assigned this Aug 17, 2026
@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 67.15686% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.59%. Comparing base (4e32927) to head (d46bbca).
⚠️ Report is 23 commits behind head on main.

Files with missing lines Patch % Lines
db/pebblev2/db.go 74.82% 23 Missing and 13 partials ⚠️
cmd/juno/dbcmd.go 40.00% 30 Missing ⚠️
node/node.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3949      +/-   ##
==========================================
- Coverage   74.99%   74.59%   -0.41%     
==========================================
  Files         446      461      +15     
  Lines       40312    41111     +799     
==========================================
+ Hits        30233    30665     +432     
- Misses       7967     8336     +369     
+ Partials     2112     2110       -2     
Flag Coverage Δ
jsonv2 77.39% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rodrodros

Copy link
Copy Markdown
Contributor

Why is there regressing values? I would expect performance to stay the same at worst?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants