MCP

scraper-farm ships an MCP endpoint over plain HTTP JSON-RPC 2.0:

• POST /mcp — per-user. Tools: scrape, analyse, map, docs, get_execution, list_executions. Use this from an LLM client to drive scrapes directly. (POST /mcp/scrape is a deprecated alias from the key-in-header era and keeps working.)

The endpoint accepts a single JSON-RPC request per POST (no SSE, no long-lived connection). Auth is OAuth (sign in with your dashboard account — no key handling) or an sf_… API key for headless use.

Claude Code (OAuth — recommended)

claude mcp add --transport http scraper-farm https://api.scrapesilo.com/mcp

Then inside Claude Code run /mcp, pick scraper-farm, and choose Authenticate. Your browser opens a sign-in with the same account you use for the dashboard; once approved, the tools connect — no API key to create, export, or rotate. Any MCP client that implements the spec’s OAuth flow (Cursor, Goose, MCP Inspector, …) discovers the same flow automatically from the endpoint’s metadata.

Scrapes run against your account’s default API key, so usage shows up in the dashboard as usual.

API key (CI / headless)

For environments where a browser sign-in isn’t possible, bearer-token auth keeps working. Add to .mcp.json in your repo (or ~/.claude/.mcp.json for global):

{
  "mcpServers": {
    "scraper-farm-scrape": {
      "type": "http",
      "url": "https://api.scrapesilo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${SCRAPER_FARM_API_KEY}"
      }
    }
  }
}

Export SCRAPER_FARM_API_KEY=sf_… in your shell (create the key under Settings → API keys).

Cursor / other clients

Cursor’s ~/.cursor/mcp.json accepts the same type: 'http' shape. So does Goose. Both the OAuth flow and the sf_… bearer work.

For clients that only support stdio MCP, run a local proxy that forwards stdio to the HTTP endpoint — mcp-remote works well and handles the OAuth flow itself (it opens the browser for you):

npx mcp-remote https://api.scrapesilo.com/mcp

(or pass --header "Authorization: Bearer $SCRAPER_FARM_API_KEY" to skip OAuth.)

Tools

Six tools. Scrape/analyse/map/executions match the REST endpoints. docs is MCP-only (catalog + markdown + fixture examples).

scrape

Same surface as POST /scrape. Args: { url, engine, actions, sessionId?, options?, useProxy?, myProxyUrl?, myProxyConfig?, solveCaptcha? } (url may be an array). engine is "html" or "browser" only. actions is required.

Returns an array of { url, data, tookMs, executionId, batchId?, error? }. Every item has executionId. batchId is set only when url was an array with more than one entry. Failed URLs: data: null + error; siblings still return.

• Deterministic: engine: "html" | "browser" + actions (see Actions DSL).
• Prefer analyse for interpretation / long-form page reading.

html — article fixture

{
  "url": "https://scrapesilo.com/fixtures/article",
  "engine": "html",
  "actions": {
    "title": "h1",
    "body": "p",
    "cta": "a.sf-cta@href"
  }
}

[
  {
    "url": "https://scrapesilo.com/fixtures/article",
    "data": {
      "title": "…",
      "body": "…",
      "cta": "https://scrapesilo.com/…"
    },
    "tookMs": 120,
    "executionId": "ex_…"
  }
]

browser — JS-rendered list

Raw HTML is empty; items appear after JavaScript. Use engine: "browser".

{
  "url": "https://scrapesilo.com/fixtures/js-list",
  "engine": "browser",
  "actions": {
    "items": {
      "selector": ".sf-item",
      "many": true,
      "output": "text"
    }
  }
}

analyse

Document analysis over page markdown (not a CSS plan). Use for outlines, summaries, yes/no flags, compliance-style reading. Args: { url, query, fetch?, options?, useProxy?, myProxyUrl?, myProxyConfig? }. Returns an array of { url, data: { result, evidence, confidence }, engine: "analyse", tookMs, executionId, batchId?, error? }.

{
  "url": "https://scrapesilo.com/fixtures/prose",
  "query": "Outline the article sections and state the main claim in one sentence."
}

map

Same surface as POST /map. Args: { url, include?, exclude?, allowSubdomains?, depth?, limit?, ignoreSitemap? } → { tookMs, sitemaps: [{ domain, url, links }] }. Unmetered. Pass a seed URL (or a sitemap .xml); discovery walks robots.txt + nested sitemap indexes.

docs

Look up documentation and the noindex fixture examples. Not a REST route.

topic	Returns
omitted	Catalog: doc slugs (with .md URLs) and fixture slugs
quickstart, engines, actions-dsl, scrape, analyse, mcp, …	That page as markdown
article, listing, nested, table, json, prose, js-list, form	Canonical scrape / analyse args against https://scrapesilo.com/fixtures/…

{ "topic": "actions-dsl" }

{ "topic": "article" }

Call this before inventing selectors or engines.

get_execution

args: { id: string } → ExecutionRow. Equivalent to GET /executions/:id.

list_executions

args: { status?, batchId?, limit? } → ExecutionRow[]. Equivalent to GET /executions. Filter by batchId to pull every row from one multi-URL scrape.

Copy for an agent

MCP URL: https://api.scrapesilo.com/mcp. Auth: OAuth (Claude Code /mcp Authenticate) or Authorization: Bearer sf_…. Tools: scrape, analyse, map, docs, get_execution, list_executions. Call docs with no topic for the catalog, or topic set to a slug (actions-dsl, article, …). scrape requires engine (html|browser) + actions. analyse requires url + query.

{
  "mcpServers": {
    "scraper-farm-scrape": {
      "type": "http",
      "url": "https://api.scrapesilo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${SCRAPER_FARM_API_KEY}"
      }
    }
  }
}

html scrape tool args:

{
  "url": "https://scrapesilo.com/fixtures/article",
  "engine": "html",
  "actions": {
    "title": "h1",
    "body": "p",
    "cta": "a.sf-cta@href"
  }
}

analyse tool args:

{
  "url": "https://scrapesilo.com/fixtures/prose",
  "query": "Outline the article sections and state the main claim in one sentence."
}