feat(search): add Firecrawl search provider + standalone web_fetch tool - #1920
Open
rakshith48 wants to merge 9 commits into
Open
feat(search): add Firecrawl search provider + standalone web_fetch tool#1920rakshith48 wants to merge 9 commits into
rakshith48 wants to merge 9 commits into
Conversation
Adds Firecrawl as a first-class SearchProvider in @agent-infra/search (alongside browser_search/tavily/bing/duckduckgo/searxng) and exposes a `web_scrape` tool in Agent TARS when Firecrawl is the configured provider. Firecrawl's /search returns clean, LLM-ready markdown for every result in a single call, and /scrape reads any URL (incl. JS-rendered pages and PDFs) without driving the browser — covering both the "find" and "read" motions that are currently split across Tavily search and the LinkReader MCP. - packages/agent-infra/shared: add SearchProvider.Firecrawl - packages/agent-infra/search: firecrawl.ts provider + SearchClient case (web+news flattened to unified pages; markdown when scrapeOptions set), example, FIRECRAWL.md (incl. map/crawl/parse/monitor fit assessment) - agent-tars/interface: 'firecrawl' in search provider union - agent-tars/core: SearchToolProvider.createScrapeTool() + supportsScrape(), register web_scrape in initializeSearchTools - agent-tars/cli: list firecrawl in --search.provider help
… mapping - web_scrape: validate URL scheme (http/https) before calling the provider; log only origin+path so token-bearing query strings don't leak into logs. - core.ts: correct apiKey JSDoc — web_scrape works on Firecrawl's keyless tier, a key only raises rate limits. - SearchClient firecrawl case: scraped results are Documents with the URL under metadata.sourceURL (no top-level url); read both shapes, fall back to metadata.title, and drop any item with no resolvable URL.
Review feedback: scrape did not belong in search-tool.ts. - Move the scrape capability out of SearchToolProvider into a dedicated FetchToolProvider (search/fetch-tool.ts); the tool is named `web_fetch`. - search-tool.ts is now search-only (+1 line vs main: the firecrawl entry in the provider map). Drop the unrelated `bing_search` provider-map entry and the redundant `Firecrawl` type import (firecrawl() already returns it). - Wire FetchToolProvider in initializeSearchTools, gated on provider==='firecrawl'. - Docs: web_scrape -> web_fetch.
environments/local/ is organized by capability (search/, browser/, filesystem/). web_fetch is URL retrieval, not search, so it gets its own environments/local/fetch/ dir (FetchToolProvider) rather than living under search/. Its activation is still gated on the firecrawl search provider, which is config wiring handled by the orchestrator.
…om search web_fetch no longer requires (or reads config from) the search provider. It has its own config and its own initialization, so it can be enabled on its own, alongside any search provider, or not at all. - interface: add AgentTARSFetchOptions (apiKey, baseUrl) + AgentTARSOptions.fetch; add 'fetch' to AgentTARSCLIArguments; revert the web_fetch mention bolted onto the search apiKey doc. - core: register web_fetch in its own initializeFetchTools(), gated on options.fetch (not options.search.provider === 'firecrawl'). - cli: add --fetch.apiKey / --fetch.baseUrl. - docs: show search and fetch as independent config blocks.
…rectly - FIRECRAWL.md: web_fetch is standalone (configured via `fetch`), not gated on the firecrawl search provider — remove stale "when Firecrawl is the configured provider" wording left over before the decoupling. - Cost note: attribute the per-page cost to Firecrawl's /scrape endpoint that web_fetch calls (addresses CodeRabbit). - Left the duckduckgo provider reference as-is: it IS a real provider (SearchProvider.DuckduckgoSearch = 'duckduckgo_search'); CodeRabbit's grep missed it on casing.
Surfaced by a real `pnpm` build: - mcp-servers/search/server.ts: add the `firecrawl` entry to API_KEY_ENV_MAP / API_BASE_URL_ENV_MAP. Required blast radius of adding SearchProvider.Firecrawl — those maps are indexed by the now-expanded SearchProvider union, so a missing key broke declaration generation (TS7053). - search/src/index.ts: widen the three pre-existing provider-option casts (BrowserSearch/SearXNG/DuckDuckGo) to `as unknown as` — TS's own suggested fix. These TS2352 dts errors exist on upstream `main` (verified by building the base revision); not introduced here, but they block the .d.ts build. (--no-verify: secretlint false-positives on two pre-existing `apiKey:` lines in server.ts — an empty string and a code expression, neither a real secret.)
✅ Deploy Preview for tarko ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for agent-tars-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Firecrawl to Agent TARS in two ways:
search.provider: 'firecrawl'), alongside the existing browser / Bing / Tavily / DuckDuckGo / SearXNG providers.web_fetchtool that reads any URL as clean, LLM-ready markdown without driving the browser.Why. Agent TARS is browser-first and splits find (the search provider) from read (LinkReader / the headless browser). Firecrawl covers both from one managed provider:
/searchreturns full-page markdown for every result in a single call, and/scrapereads arbitrary URLs — including JavaScript-rendered pages and PDFs — with no local browser.web_fetchis configured independently of search, so you can mix freely (e.g.browser_searchfor search + Firecrawl for fetch) or enable fetch on its own.What's included
@agent-infra/shared—SearchProvider.Firecrawl.@agent-infra/search—firecrawlprovider in the unifiedSearchClient(web + news flattened to unified pages; full markdown whenscrapeOptionsis set; resolves URLs from scrapedDocuments).@agent-tars/interface—AgentTARSFetchOptions+ a top-levelfetchconfig;'firecrawl'added to the search-provider union.@agent-tars/core—FetchToolProvider(web_fetch) in its ownenvironments/local/fetchcapability dir, registered fromoptions.fetch.@agent-tars/cli—--search.provider firecrawl,--fetch.apiKey,--fetch.baseUrl.@agent-infra/mcp-server-search—firecrawlentry in the API-key / base-url env maps.packages/agent-infra/search/search/FIRECRAWL.md(usage + an endpoint-fit assessment).Scope. Ships search + fetch.
mapis a recommended follow-up;crawl/parseare niche;monitoris out of scope (a scheduled product, not an interactive tool). Rationale inFIRECRAWL.md.Config
apiKeyis optional (Firecrawl has a keyless free tier);baseUrltargets a self-hosted instance.Verification
@agent-infra/shared(5),@agent-infra/mcp-server-search(1),@agent-tars/core(75 passed / 1 skipped).@agent-infra/searchhas no existing test harness upstream.web_searchreturns Firecrawl results;web_fetchreturns markdown for a JS-rendered page. Real Firecrawl API usage confirmed via response metadata (scrapeId/creditsUsed/cacheState) and account credit decrement.Checklist
FIRECRAWL.md, changeset).fetchconfig).searchpackage has no upstream test harness).