Giving an Agent a Search Engine It Actually Owns

Wire a coding agent up to a third-party model — Kimi K2 via OpenRouter, DeepSeek through your own Azure endpoint, Qwen on a vLLM box, anything behind a proxy or aggregator — and a quiet thing happens to its toolbelt. The WebSearch tool that ships with Claude Code is bound to the official Anthropic backend, so the moment you point Claude Code at an OpenAI-compatible endpoint, it silently disappears. What’s left is WebFetch — “give me a URL and I’ll read the HTML.”

WebFetch is fine for one page. It is miserable for “look this up.” There is no discovery. There is no ranking. There is no way to fan out across ten candidate sources and pick the best three. You are back to copy-pasting URLs by hand, which is the exact thing you brought the agent in to avoid.

The fix I keep coming back to is also the cheapest one: run your own search engine. Specifically, self-host SearXNG for discovery and Firecrawl for extraction, both as small Docker stacks on a box you control — and, as I’ll argue below, a box at home beats a VPS. Your agent gets a real search back, you keep the queries off third-party infrastructure, and the whole thing sets up by asking your favourite agent to do it for you.

The shape of the problem

There are three jobs a research-capable agent has to do, and most tools do exactly one of them:

When WebSearch is unavailable and you only have WebFetch, you collapse Discover + Extract into one step that can only start once you already know the URL — which is the entire problem. The agent has to either ask you for URLs or hallucinate them, and neither is acceptable for a research task.

The SearXNG + Firecrawl split restores all three jobs without depending on any vendor:

• SearXNG is an open-source meta-search engine. It can federate over 200 upstream engines — you switch on the handful you want (Bing, Mojeek, DuckDuckGo, Startpage, Brave, …) — dedupes the results, and returns them as JSON over a tiny HTTP API. You self-host it. No accounts, no keys, no rate-limit contract.
• Firecrawl is an open-source scraper that turns a URL into clean Markdown — or, with a JSON schema, into a typed object. It handles JavaScript-heavy pages, waits for content, and follows links. The hosted version has a generous free tier; the self-hosted one is a small docker compose stack (API, worker, Redis, a headless-browser service) that costs nothing per page to scrape.

Together they replace the missing WebSearch and upgrade the WebFetch you already have. The agent can now do the thing you actually wanted: ask a question, get ranked URLs, read the best ones, cite them.

Why this combo is the right default

I have run a lot of variations on this — Brave Search API, SerpAPI, Tavily, hosted Firecrawl, headless Chrome — and I keep landing back on self-hosted SearXNG + self-hosted Firecrawl for five specific reasons.

1. Privacy and provenance. Every search and every scrape originates from your own box, on your own egress, with no account attached. The upstream engines still see the query text — that’s unavoidable for any search — but no aggregator ties it to a profile, monetises it, or files it under your name, and the pages you scrape land on your server instead of a vendor’s logs. For research on unpublished work, internal docs, or anything you would rather not see in someone else’s training set, that is the honest version of private.

2. No rate limits, no bill. The hosted Firecrawl free tier is generous right up until it isn’t. Self-hosting flips the cost model: a box you already own — or a $5 VPS — scrapes more pages in an afternoon than a free-tier account gives you in a month.

3. Structured extraction. Hosted Firecrawl’s killer feature is JSON-mode with a schema. Self-hosted Firecrawl supports the same "formats":["json"] + "jsonOptions":{"schema":{...}} path. Typed fields out of a pricing page, a model catalogue, an arXiv abstract — the agent gets back exactly the fields it asked for, not 8k tokens of Markdown to grep. This is the single biggest token saving in my whole stack.

One caveat the marketing skips: JSON-with-schema runs an LLM under the hood, so self-hosted Firecrawl needs an LLM endpoint for that mode (OPENAI_API_KEY, or any OpenAI-compatible / Ollama / vLLM URL). Plain Markdown scraping stays keyless. And there’s a privacy wrinkle: the extractor sees the full scraped page, so pointing it at a hosted API quietly hands those contents to a third party — exactly what self-hosting was meant to avoid. Keep the extraction model local (Ollama, or your own vLLM box) and you get both: per-page cost rounds to zero, and the privacy guarantee holds end to end.

4. Parallel-friendly. SearXNG returns JSON. Firecrawl is a single POST /v1/scrape. Both compose trivially across many sub-agents: 5 finders in parallel, 5 scrapers in parallel, one synthesiser. The vendor-hosted equivalents either serialise behind rate limits or charge per parallel worker.

5. The agent can run it. Because SearXNG and Firecrawl are both boring docker compose stacks behind plain HTTP APIs, an agent can install, configure, recover, and debug them with the same muscle it already has for everything else. You don’t need a DevOps person; you need a coding agent and a box to put them on.

The setup is one prompt

This is the part I keep surprising people with. I am not going to give you a 40-step Docker tutorial, because you shouldn’t be reading one. Your agent should be. Paste this — adapted to your own infra — into Claude Code, Codex, OpenCode, or whatever you drive, and let it drive the install:

I have a small always-on Linux box at home I’ll SSH into (a mini-PC, NAS, or Raspberry Pi — the residential IP is the whole point). Install two Docker stacks on it: SearXNG (meta-search, exposes :8080) and Firecrawl (scraper, exposes :3002). Both bind to localhost only — I’ll reach them over Tailscale, never the public internet. Enable Bing, Mojeek, DuckDuckGo, and Startpage in SearXNG, and turn on its JSON output format. Firecrawl needs no API key for plain Markdown scraping; if you also wire up JSON-schema extraction, point it at a local model (an Ollama box, or a vLLM endpoint on the same tailnet) so scraped page content never leaves my own infrastructure. After install, give me a 5-line smoke test I can run from my laptop to confirm both endpoints are up and returning real results.

That is the entire tutorial. The agent will SSH in, write the docker-compose.yml files, pull the images, start them, write a SearXNG settings.yml with sane defaults, and hand you the smoke-test script. No always-on machine at home? A VPS still works — Oracle’s free tier, or a Hetzner CX22 at ~€4/month — but read the next section first, because a datacenter IP quietly changes the engine math.

The bits worth being explicit about, because they are the only parts the agent will get wrong if you leave them to defaults:

• Bind to localhost, reach it over Tailscale. ports: ["127.0.0.1:8080:8080"], not "8080:8080". Put a Tailscale daemon on the box and hit the endpoints as http://<box-tailnet-name>:8080 and http://<box-tailnet-name>:3002 from your laptop and your agents. This is also what makes a home box no harder to run than a VPS: Tailscale punches through NAT, so the box never needs a public IP or a forwarded port — just outbound internet. Don’t expose the service publicly; let the tailnet be the only door.
• Host at home and the engine math gets easy — this is the real reason I moved off a VPS. Search engines treat datacenter IPs as presumed-guilty. From a hyperscaler range (AWS, GCP, the big Hetzner/OVH pools) SearXNG starts handing back empty results within a handful of queries; from a residential IP you look like a person, and Bing, Mojeek, DuckDuckGo, and Startpage mostly just work. If you’re stuck on a VPS, budget for a fight: Bing and Mojeek usually egress direct, DuckDuckGo and Startpage often need a residential proxy, and Google’s native engine 403s almost every time (Startpage covers Google’s index, so the loss is small). I ran the datacenter version for months and ended up tunnelling half my engines through a proxy on a Raspberry Pi at home — at which point the Pi may as well host the whole thing.
• Turn on SearXNG’s JSON format and confirm it. format=json is disabled by default, and it’s the difference between a parseable API and a wall of HTML. Add json to search.formats in settings.yml, then test it the moment the container starts; do not assume.
• Firecrawl inherits the box’s IP too — leave PROXY_SERVER off until you need it. Same logic as the engines: from home, far fewer sites hard-block you. On a VPS, sites like LinkedIn or some paywalled news pass SearXNG discovery and then fail the Firecrawl fetch from the datacenter IP — that’s the moment to wire a residential proxy into Firecrawl’s env. Until you hit one, leave it off; it slows every page and is one more thing that can break.

What the agent actually does with it

Once both containers are up, the loop is small enough to fit on an index card. Here is the discovery half:

1
2
3
4
5
6
7
# 1. Discover: turn the question into 3–8 ranked URLs
Q=$(python3 -c "import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1]))" \
    "Claude Opus 4.8 pricing per million tokens")
curl -s "http://<your-searxng>:8080/search?q=${Q}&format=json" \
  | python3 -c "import sys,json; \
d=json.load(sys.stdin); \
print('\n'.join(f\"{r['title']}\\t{r['url']}\" for r in d['results'][:8]))"

And the extraction half, in JSON mode with a schema — this is the pattern that earns its keep on structured pages:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 2. Extract: typed fields out of a pricing page
curl -s -X POST http://<your-firecrawl>:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["json"],
    "jsonOptions": {
      "schema": {
        "type": "object",
        "properties": {
          "model":      {"type": "string"},
          "input_per_m": {"type": "number"},
          "output_per_m":{"type": "number"}
        }
      }
    }
  }' | python3 -m json.tool

For whole docs sites, use /v1/crawl once with a limit instead of paginating via search. For pages that fail (JS-heavy, login-walled, datacenter-blocked), the fallbacks in order are: web_fetch → a headless browser the agent can drive → asking you to attach a tab in your own Chrome. The fallbacks are cheaper than they sound because the discovery step already filtered out 90% of the candidates, so you only fall back on the 1–2 you actually need.

I call it a loop, not a pipeline, because of the triage step in the middle: the agent should read the SearXNG titles and snippets before it commits to scraping anything. A snippet that already answers the question needs no fetch. A URL that looks off-topic needs no fetch. This one rule keeps the loop fast and cheap. Skip it, and the agent reflexively scrapes every result — burning the whole token budget on a page that turned out to be a blog post from 2021.

The version of this that lives in my skill

I packaged the loop above into a small Claude Code skill — a SKILL.md the orchestrating model reads, plus a fallback ladder and a “lanes in order” rule (official docs → SERP refine → community → arXiv). The skill tells the agent:

1. Up-check both endpoints first. One-line curl per service. If either is down, recover (restart the container) before doing anything else.
2. Triage before fetching. Pick 3–8 URLs from SearXNG, not 30.
3. Prefer JSON-mode Firecrawl with a schema for any structured page. The token saving is large enough that the agent should default to it.
4. Always end with a Sources section of Markdown links. Pricing/spec claims tagged as of <DATE>.
5. Facts from official sources; community as signal, never as proof. Flag weak or conflicting evidence instead of silently averaging it.

The interesting failure mode the skill prevents is the “research hallucination” — the agent confidently cites a URL it never actually read, because the default tools make it easy to fabricate citations when the search box is missing. With a real search endpoint the agent has to find URLs before it can cite them, and the discipline of a Sources section forces it to list what it actually used.

The two takeaways

If your agent can’t search, it can’t research. A coding agent pointed at a non-Anthropic backend loses more than WebSearch — it loses the entire “look this up” pattern, because WebFetch only works on URLs you already have. Restoring search restores the loop.

The cheapest way to restore it is two Docker stacks you control. SearXNG and Firecrawl are open-source, containerised, and run happily on a spare box at home — and a home IP is what keeps the search engines from CAPTCHA-walling you. They are private, unmetered, parallel-friendly, and structured-extraction-capable in a way the vendor-hosted alternatives mostly are not. And the install is genuinely one prompt to your favourite coding agent — there is no reason for a human to read a Docker tutorial in 2026 when an agent will write one that is exact to your box.

The setup is the smallest part. Once the endpoints are up, the way your agent does research changes: it stops asking you for URLs and starts coming back with a real Sources section, tagged dates, and typed fields pulled straight out of the pages it actually read.

Jingbiao Mei is a final-year PhD student at the University of Cambridge’s Machine Intelligence Lab, working on multimodal retrieval, agent systems, and the kind of tool-usage problems that only show up after the third coding agent is on the machine at the same time. He is the creator of Tokdash, ATM-Bench, FLMR, PreFLMR, and ExPO-HM.

• Previous
  The ATM-Bench Leaderboard is Live
• Next
  Your Agent's Work Belongs on a Web Page