CNDB-18906: Implement a trie serialization format for partition updates - #2572
CNDB-18906: Implement a trie serialization format for partition updates#2572lesnik2u wants to merge 76 commits into
Conversation
Implements a row-level trie memtable that uses deletion-aware tries to store deletions separately from live data, together with the associated TrieBackedPartition and TriePartitionUpdate. Refactors trie hierarchy to support multiple trie types: - plain - range, which stores range boundaries and is able to answer questions about the range that applies to every point in the trie - deletion aware, which combines a data part and a deletion range trie Every trie type supports suitable operations, including merging and intersection that make sense for the type of trie. In particular, deletion-aware tries apply range branches to delete data during merges. Adds a new method to UnfilteredRowIterator that is implemented by the new trie-backed partitions to ask them to stop issuing tombstones. This is done on filtering (i.e. conversion from UnfilteredRowIterator to RowIterator) where tombstones have already done their job and are no longer needed. Adds JMH tests of tombstones that demonstrate tombstone-independent performance on memtable queries. # Conflicts: # test/burn/org/apache/cassandra/index/sai/LongVectorTest.java
in a combined `encodedState` returned by advancing methods. This saves megamorphic calls to `incomingTransition` and can be augmented by further information at no cost.
This functionality has two main applications: - it allows reverse walks that present prefix content in the correct byte-comparable order (i.e. prefixes after children) - it makes it possible to have full control over what is and isn't included in a trie ranges (e.g. making it possible to have a branch set and nested ranges)
…and TrieMemtable to Stage3 version Remove duplicate configuration object and add tests for stage 3
This change extends the coverage of the memtable trie to the cell level, defining mappings of trie branches to and from the legacy concepts of complex columns and rows.
This makes it possible to have completely off-heap trie memtable, where cell data is stored inside the trie structure if it is small enough to fit, or placed in natively-allocated memory and referenced by memory address.
Drops the mutator tail retrieval methods which are no longer necessary
- Memtable will now request a switch if columns change - It will use the metadata state at construction to make sure it is not affected by breaking changes in columns - It will use current metadata as source of dropped columns
The main reason for this is that serialized data may be using other columns than the ones in the metadata (e.g. synthetic columns) and we don't yet have support for changing column sets. We don't gain anything either because we can't do merges by trie yet.
Because the updates are serialized (for the commitlog and replicas) and the serialization code relies heavily on the legacy access mechanisms in a way that cannot be easily optimized, using trie updates causes significant overhead. The update is converted to trie in MemtableShard.put. This decision will be changed when we introduce a trie-based version of the partition serialization.
5cbc89d to
8cc62a6
Compare
|
Something looks wrong in the first benchmark table. Legacy memtable partition update serialization should not be affected by the new version as they are running exactly the same code. In the third one, did you run the benchmark with |
|
It's an interesting approach to dump and restore the in-memory state. Does it pass most tests? If it does, I will run the fallout benchmark to see what effect it has. Ultimately we should be using an on-disk format, non-page-packed, like the one I started developing in this commit of the trie-table branch. Give me a few days to add deletion-aware support and to make sure we can build both page-packed (for trietables) and faster non-packed (for commitlog/messaging) serializations. |
|
@blambov The differences were because In C* 5.0 (VERSION_DS_20), Cassandra introduced major internal optimizations to standard row and cell serialization. Because of these general 5.0 protocol optimizations, all Legacy B-Tree updates are ~10x faster to serialize/deserialize under VERSION_DS_20 than VERSION_DS_10. |
|
Shouldn't we then use a new version, as we are changing the serialization? |
|
@blambov yeah, I was intending to add a new one. All trie tests pass for me, and all of the commit log tests pass |
lesnik2u
changed the title
2f6bbc5 to
95fe2db
Compare
2 participants
What is the issue
https://github.com/riptano/cndb/issues/18906
What does this PR fix and why was it fixed
...