Every paid crawl now reaches the people whose niche was crawled - #14
Merged
Conversation
Phase 4. The gateway has been selling crawl passes since the site launched and the money went two places: crawl_sales, and the partner split. It never reached the revenue ledger, so a Knowledge Influencer operating a niche earned nothing from a crawler paying to read it. The hard part is that a pass buys the whole index for a day, not one niche, so there is no single niche to hand it to. The only measurable answer to whose data it paid for is how much of the index each niche holds, which is the rule the partner split already uses. One sale therefore becomes several revenue events: one per operated niche, sized pro-rata against the WHOLE index, plus an unattributed one for the part nobody operates. Against the whole index deliberately. A niche holding one percent of the rows must not collect the whole dollar because it happens to be the only one with an operator. Every cent lands somewhere. The largest-remainder split that already kept basis points summing to 8000 is now a general `apportion`, used by both, so the parts sum to exactly the sale rather than to 99 cents. The unoperated remainder is a claimant in that division rather than a leftover. Each event records the item counts its share was computed from, so the number can be argued with. Idempotent per event (`x402:<ref>:<niche>` and `x402:<ref>`), so a settlement delivered twice books once even though it lands as several rows. Never allowed to fail the sale: the money has already moved. The splitting arithmetic lives in packages/knowledge with the rest of the pure domain rather than beside the database call, which is also what lets it be tested without a DATABASE_URL. Verified by calling the real onSale hook: a $1 sale over an index of 100 items with games (90) operated and research (10) not booked 90c to games and 10c unattributed, the operator was owed 18c at the contributor rate, redelivery changed nothing, and once research had an operator too the next sale split 90/10 with no remainder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W9NGGLDvNayi6uWheSXDGF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 4: the gateway's
onSalehook now books into the revenue ledger, so a paid crawl reaches the people operating the niches it read.What was actually missing
Worth stating plainly, because it is easy to assume this already worked. The gateway has been selling crawl passes since launch, and
onSalealready did two things: recorded the sale incrawl_sales, and split it to partners. It never touched the revenue ledger — so a Knowledge Influencer operating a niche earned nothing when a crawler paid to read it. That is the gap this closes.The hard part
A pass buys the whole index for a day, not one niche. There is no single niche to attribute it to.
The only measurable answer to "whose data did this pay for" is how much of the index each niche holds — the same rule the partner split already uses. So one sale becomes several revenue events:
Against the whole index deliberately: a niche holding one percent of the rows must not collect the whole dollar just because it is the only one with an operator.
Every cent lands somewhere
The largest-remainder split that already kept basis points summing to 8000 is now a general
apportion, used by both. The unoperated remainder is a claimant in that division rather than a leftover, so the parts sum to exactly the sale rather than to 99 cents. Each event records the item counts its share was computed from, so the number can be argued with rather than believed.Idempotent per event (
x402:<ref>:<niche>andx402:<ref>) — a settlement delivered twice books once even though it lands as several rows. Attribution is never allowed to fail the sale; the money has already moved.One structural change
splitSaleAcrossNicheslives inpackages/knowledgewith the rest of the pure domain, not beside the database call. That is also what lets it be tested without aDATABASE_URL— the first version imported the db module and broke the suite's "needs no server" property.Verification
bun test— 177 pass, 0 fail (12 new). Needs no server.bun run lint— clean; the 4 CSS warnings andbiome.jsondeprecation are pre-existing onmain.onSalehook, not a mock: a $1 sale over an index of 100 items withgames(90) operated andresearch(10) not booked 90c to games and 10c unattributed; the operator was owed 18c at the contributor rate; redelivering the identical sale left the ledger unchanged; and onceresearchhad an operator too, the next sale split 90/10 with no remainder.crawl_salesstill records exactly as before.Worth knowing before this matters
There have been zero crawl sales on nichedb.dev so far — the paywall is live and returns 402, but nothing has bought a pass yet, and there are no partner accounts or niche operators either. This wires the pipe before anything flows through it, which is the right order: retroactively attributing sales that were never recorded against a niche is much harder than attributing them as they arrive.
🤖 Generated with Claude Code
https://claude.ai/code/session_01W9NGGLDvNayi6uWheSXDGF