---
title: MCP
description: Wire scraper-farm into Claude Code, Cursor, Goose, or any Model-Context-Protocol-aware client.
---

scraper-farm ships an MCP endpoint over plain HTTP JSON-RPC 2.0:

- **`POST /mcp`** — per-user. Tools: `scrape`, `analyse`, `map`, `docs`, `get_execution`, `list_executions`. Use this from an LLM client to drive scrapes directly. (`POST /mcp/scrape` is a deprecated alias from the key-in-header era and keeps working.)

The endpoint accepts a single JSON-RPC request per POST (no SSE, no long-lived connection). Auth is OAuth (sign in with your dashboard account — no key handling) or an `sf_…` API key for headless use.

## Claude Code (OAuth — recommended)

```bash
claude mcp add --transport http scraper-farm https://api.scrapesilo.com/mcp
```

Then inside Claude Code run `/mcp`, pick **scraper-farm**, and choose **Authenticate**. Your browser opens a sign-in with the same account you use for the dashboard; once approved, the tools connect — no API key to create, export, or rotate. Any MCP client that implements the spec's OAuth flow (Cursor, Goose, MCP Inspector, …) discovers the same flow automatically from the endpoint's metadata.

Scrapes run against your account's default API key, so usage shows up in the dashboard as usual.

## API key (CI / headless)

For environments where a browser sign-in isn't possible, bearer-token auth keeps working. Add to `.mcp.json` in your repo (or `~/.claude/.mcp.json` for global):

```json
{
  "mcpServers": {
    "scraper-farm-scrape": {
      "type": "http",
      "url": "https://api.scrapesilo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${SCRAPER_FARM_API_KEY}"
      }
    }
  }
}
```

Export `SCRAPER_FARM_API_KEY=sf_…` in your shell (create the key under Settings → API keys).

## Cursor / other clients

Cursor's `~/.cursor/mcp.json` accepts the same `type: 'http'` shape. So does Goose. Both the OAuth flow and the `sf_…` bearer work.

For clients that only support `stdio` MCP, run a local proxy that forwards stdio to the HTTP endpoint — `mcp-remote` works well and handles the OAuth flow itself (it opens the browser for you):

```bash
npx mcp-remote https://api.scrapesilo.com/mcp
```

(or pass `--header "Authorization: Bearer $SCRAPER_FARM_API_KEY"` to skip OAuth.)

## Tools

Six tools. Scrape/analyse/map/executions match the REST endpoints. `docs` is MCP-only (catalog + markdown + fixture examples).

### `scrape`

Same surface as [`POST /scrape`](/docs/api/scrape). Args: `{ url, engine, actions, sessionId?, options?, useProxy?, myProxyUrl?, myProxyConfig?, solveCaptcha? }` (`url` may be an array). `engine` is `"html"` or `"browser"` only. `actions` is required.

Returns an **array** of `{ url, data, tookMs, executionId, batchId?, error? }`. Every item has `executionId`. `batchId` is set only when `url` was an array with more than one entry. Failed URLs: `data: null` + `error`; siblings still return.

- **Deterministic:** `engine: "html"` | `"browser"` + `actions` (see [Actions DSL](/docs/actions-dsl)).
- Prefer **`analyse`** for interpretation / long-form page reading.

#### html — article fixture

```json
{
  "url": "https://scrapesilo.com/fixtures/article",
  "engine": "html",
  "actions": {
    "title": "h1",
    "body": "p",
    "cta": "a.sf-cta@href"
  }
}
```

```json
[
  {
    "url": "https://scrapesilo.com/fixtures/article",
    "data": {
      "title": "…",
      "body": "…",
      "cta": "https://scrapesilo.com/…"
    },
    "tookMs": 120,
    "executionId": "ex_…"
  }
]
```

#### browser — JS-rendered list

Raw HTML is empty; items appear after JavaScript. Use `engine: "browser"`.

```json
{
  "url": "https://scrapesilo.com/fixtures/js-list",
  "engine": "browser",
  "actions": {
    "items": {
      "selector": ".sf-item",
      "many": true,
      "output": "text"
    }
  }
}
```

### `analyse`

Document analysis over page markdown (not a CSS plan). Use for outlines, summaries, yes/no flags, compliance-style reading. Args: `{ url, query, fetch?, options?, useProxy?, myProxyUrl?, myProxyConfig? }`. Returns an **array** of `{ url, data: { result, evidence, confidence }, engine: "analyse", tookMs, executionId, batchId?, error? }`.

```json
{
  "url": "https://scrapesilo.com/fixtures/prose",
  "query": "Outline the article sections and state the main claim in one sentence."
}
```

### `map`

Same surface as [`POST /map`](/docs/api/map). Args: `{ url, include?, exclude?, allowSubdomains?, depth?, limit?, ignoreSitemap? }` → `{ tookMs, sitemaps: [{ domain, url, links }] }`. Unmetered. Pass a seed URL (or a sitemap `.xml`); discovery walks robots.txt + nested sitemap indexes.

### `docs`

Look up documentation and the noindex fixture examples. Not a REST route.

| `topic` | Returns |
|---|---|
| omitted | Catalog: doc slugs (with `.md` URLs) and fixture slugs |
| `quickstart`, `engines`, `actions-dsl`, `scrape`, `analyse`, `mcp`, … | That page as markdown |
| `article`, `listing`, `nested`, `table`, `json`, `prose`, `js-list`, `form` | Canonical `scrape` / `analyse` args against `https://scrapesilo.com/fixtures/…` |

```json
{ "topic": "actions-dsl" }
```

```json
{ "topic": "article" }
```

Call this before inventing selectors or engines.

### `get_execution`

`args: { id: string }` → `ExecutionRow`. Equivalent to [`GET /executions/:id`](/docs/api/executions).

### `list_executions`

`args: { status?, batchId?, limit? }` → `ExecutionRow[]`. Equivalent to [`GET /executions`](/docs/api/executions). Filter by `batchId` to pull every row from one multi-URL scrape.

## Copy for an agent

MCP URL: `https://api.scrapesilo.com/mcp`. Auth: OAuth (Claude Code `/mcp` Authenticate) or `Authorization: Bearer sf_…`. Tools: `scrape`, `analyse`, `map`, `docs`, `get_execution`, `list_executions`. Call `docs` with no topic for the catalog, or `topic` set to a slug (`actions-dsl`, `article`, …). `scrape` requires `engine` (`html`|`browser`) + `actions`. `analyse` requires `url` + `query`.

```json
{
  "mcpServers": {
    "scraper-farm-scrape": {
      "type": "http",
      "url": "https://api.scrapesilo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${SCRAPER_FARM_API_KEY}"
      }
    }
  }
}
```

html scrape tool args:

```json
{
  "url": "https://scrapesilo.com/fixtures/article",
  "engine": "html",
  "actions": {
    "title": "h1",
    "body": "p",
    "cta": "a.sf-cta@href"
  }
}
```

analyse tool args:

```json
{
  "url": "https://scrapesilo.com/fixtures/prose",
  "query": "Outline the article sections and state the main claim in one sentence."
}
```