Repository navigation
badger, lmdb: match tag values exactly, and check every tag of a filter - #75
Open
sunmoonron wants to merge 1 commit into
Open
sunmoonron wants to merge 1 commit into
sunmoonron wants to merge 1 commit into
Conversation
sunmoonron
force-pushed
the
tag-exact-match
branch
from
September 8, 2026 16:32
69b9a77 to
3f50c92
Compare
For plain string tag values, and for the d part of a-style values, the badger tag index key is the value itself with nothing after it, so a query for d=a also walks the keys of a-b, abc and so on, and badger dropped the indexed tag from the post-fetch check, trusting the index. The visible symptom is khatru#52: with a-b stored first and newer, publishing an addressable event with d=a was silently not stored, because the replace query for d=a matched a-b. badger keeps the indexed tag in the extra filter, so the fetched event is checked for the exact value. lmdb's keys are exact for these cases, but when a filter has authors or kinds and two tags the planner has no index preference for, only one of the tags was enforced. It now checks every tag of the filter against the fetched event, in queries and in both count paths. A shared regression test covers d, a, t and unknown tags; without the change it fails on badger at the d=a query and on lmdb at the two unknown tags query. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sunmoonron
force-pushed
the
tag-exact-match
branch
from
September 8, 2026 16:33
3f50c92 to
aee2f62
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the bug behind fiatjaf/khatru#52 (the "silent rejection of certain replaceable events"), plus a smaller lmdb gap found while writing the test.
badger. For plain string tag values, and for the
dpart ofa-style values, the tag index key is the value itself with nothing after it, so a prefix iteration ford=aalso walks the keys ofa-b,abcand so on. The planner then dropped the indexed tag from the post-fetchextraFilter(CopyMapWithoutKey), trusting the index to be exact. So witha-bstored first and newer, publishing an addressable event withd=awas silently not stored: the replace query ford=areturned thea-bevent,IsOldersaid the new one was older, andshouldStorewent false with no error. Plain tag queries had the same false positives. Fix: the indexed tag stays inextraFilter.Tags, sofilterMatchesTagschecks the fetched event for the exact value. No index format change, no migration.lmdb. Its keys are already exact for these cases, so it did not have the
d=abug. But when a filter has authors or kinds and two tags the planner has no index preference for (thedefaultbranch ofChooseNarrowestTag),prepareQueriesreturned only one extra tag and the other was never enforced. It now returns the wholeTagMap, and the query and both count paths check every tag against the fetched event (tagsMatch).test.
tagPrefixTestruns on every backend fromtest/db_test.go:d=anext tod=a-bthroughRelayWrapper.Publish,atags pointing at those addresses, plainttags, a two-condition filter, and a two-unknown-tags filter. Without the change it fails on badger at thed=aquery and on lmdb at the two-unknown-tags query; with it,TestSliceStore,TestBadgerandTestLMDBpass.Cost: one tag scan per fetched candidate on tag queries, which were already decoding the event.
A length-prefixed or terminated badger index key would remove the false positives at the iterator level instead; that needs a migration, so it is left out of this PR, happy to follow up if you prefer that shape.
🤖 Generated with Claude Code