A web index for AI, built through open competition on Bittensor.
AI applications need relevant, current sources and text they can use, soon enough to finish their task. Desearch collects and refreshes useful pages from the public web, prepares them for retrieval and develops the models that help AI find the right sources. Search is the first application: fast search for AI agents, and deeper research for questions that take longer.
The work happens in the open on Bittensor Subnet 22. Miners build and refresh the collection, validators check their work under published rules, and the Desearch team integrates accepted work into a tested index and the search API.
⛏️ Mine · 🛡️ Validate · ⚙️ How it works · 🧭 Desearch 2.0 · 🔑 Search API
In Desearch 1.0, miners answered search requests and validators judged the answers. It worked, but good answers could arrive too late, some depended on outside search providers, and little of the work left anything that could be kept and improved.
Desearch 2.0 prepares information before a query arrives. Pages are collected, cleaned, deduplicated, organized and prepared for retrieval in the background, so a query searches a maintained collection instead of starting from scratch. That gives:
- Faster retrieval: expensive preparation moves out of the query path
- More control over quality: coverage, freshness and supporting text can be inspected, and models improved against the same collection
- Work that lasts: every accepted page serves many later searches, not one answer
The collection covers the public web: company websites, news, articles, blogs and documentation. It starts with selected sources and grows as results prove useful.
| Work | Who does it | Status |
|---|---|---|
| Crawl and refresh pages, extract their text | Miners fetch assigned pages; validators re-check a sample of their tasks | Running on SN22 |
| Embed accepted text with a selected model | Miners run the model on their GPUs; validators recompute a sample | Running on SN22, with Desearch's embedding model |
| Train retrieval models, improve ranking, extraction and coverage | Model builders and data operators | Later programs, announced before they open |
| Build, test and serve the index | The Desearch team | Team-operated; queries never wait on miner work |
Miners crawl and embed: they take tasks, fetch the assigned pages and upload the extracted text, and turn that text into vectors on their own GPU. They are paid by their share of accepted work, and can test their miner locally before registering. What a miner does → · Miner setup → · Test locally → · Emission →
Validators check miners' work, the pages they crawl and the vectors they embed, and set weights from the work that holds up. What a validator does → · Validator setup →
The rules are published and every result is public. How it works →
The search API is available today for developers building AI agents and research tools. Get a key in the console and start with the documentation.
| SDK | Install |
|---|---|
| Python | pip install desearch-py |
| JavaScript / TypeScript | npm install desearch-js |
| MCP server | npm install -g desearch-mcp-server |
| Guide | What's inside |
|---|---|
| ⛏️ Miner | What a miner does and how it earns; setup: install, register, configure, run, monitor |
| 🧪 Test your miner locally | Run your miner against a local task API and validator before registering |
| 🛡️ Validator | What a validator does: checks and weights; setup: install, register, configure, run, monitor |
| 💰 Emission | How miners' shares are worked out and what raises them |
| ⚙️ Architecture | How the bot, task API, miners, validators and storage work together |
| 🧠 Embedding tasks | The embedding model, what a task contains and how it is checked |
| 🔌 Task API | Endpoints, rounds, scoring rules, public logs and the storage layout |
| 🧭 Desearch 2.0 | The direction: why an index, the first phase, incentives and participation |
- Discord: questions, mining and validating support
- X and Telegram: announcements
- Blog: guides and engineering write-ups
Released under the MIT License.