MCP

MCP URL: `https://api.scrapesilo.com/mcp`. Auth: OAuth (Claude Code `/mcp` Authenticate) or `Authorization: Bearer sf_…`. Tools: `scrape_markdown`, `scrape_html`, `scrape_json`, `scrape_browser`, `scrape_ai`, `extract_google_ads`, `extract_facebook_ads`, legacy `scrape`, `analyse`, `map`, `docs`, `get_job`, `list_jobs`, `get_job_result`, `list_job_results`. Call `docs` with no topic for the catalog, or `topic` set to a slug (`actions-dsl`, `article`, …). `scrape` requires `engine` (`html`|`json`|`browser`); html/browser require `actions`, json allows omitted or empty actions. `analyse` requires `url` + `query`. Parse tool text as `{ ok, data }` or `{ ok: false, error }`; unfinished submissions return `data.jobs`. Poll those IDs without resubmitting.

```json
{
  "mcpServers": {
    "scrapesilo": {
      "type": "http",
      "url": "https://api.scrapesilo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${SCRAPESILO_API_KEY}"
      }
    }
  }
}
```

For page reading, call `scrape_markdown`:

```json
{ "url": "https://scrapesilo.com/fixtures/article" }
```

For named fields, call `scrape_html`:

```json
{
  "url": "https://scrapesilo.com/fixtures/article",
  "actions": {
    "title": "h1",
    "body": "p",
    "cta": "a.sf-cta@href"
  }
}
```

analyse tool args:

```json
{
  "url": "https://scrapesilo.com/fixtures/prose",
  "query": "Outline the article sections and state the main claim in one sentence."
}
```

Markdown agents.txt

ScrapeSilo ships an MCP endpoint over plain HTTP JSON-RPC 2.0:

POST /mcp — per-user. Tools: scrape_markdown, scrape_html, scrape_json, scrape_browser, scrape_ai, extract_google_ads, extract_facebook_ads, legacy scrape, analyse, map, docs, get_job, list_jobs, get_job_result, list_job_results. Use this from an LLM client to drive scrapes directly. (POST /mcp/scrape is a deprecated alias from the key-in-header era and keeps working.)

The endpoint accepts a single JSON-RPC request per POST (no SSE, no long-lived connection). Auth is OAuth (sign in with your dashboard account — no key handling) or an sf_… API key for headless use.

Claude Code (OAuth — recommended)

claude mcp add --transport http scrapesilo https://api.scrapesilo.com/mcp

Then inside Claude Code run /mcp, pick scrapesilo, and choose Authenticate. Your browser opens a sign-in with the same account you use for the dashboard; once approved, the tools connect — no API key to create, export, or rotate. Any MCP client that implements the spec’s OAuth flow (Cursor, Goose, MCP Inspector, …) discovers the same flow automatically from the endpoint’s metadata.

Scrapes run against your account’s default API key, so usage shows up in the dashboard as usual.

API key (CI / headless)

For environments where a browser sign-in isn’t possible, bearer-token auth keeps working. Add to .mcp.json in your repo (or ~/.claude/.mcp.json for global):

{
  "mcpServers": {
    "scrapesilo": {
      "type": "http",
      "url": "https://api.scrapesilo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${SCRAPESILO_API_KEY}"
      }
    }
  }
}

Export SCRAPESILO_API_KEY=sf_… in your shell (create the key under Settings → API keys).

Cursor / other clients

Cursor’s ~/.cursor/mcp.json accepts the same type: 'http' shape. So does Goose. Both the OAuth flow and the sf_… bearer work.

For clients that only support stdio MCP, run a local proxy that forwards stdio to the HTTP endpoint — mcp-remote works well and handles the OAuth flow itself (it opens the browser for you):

npx mcp-remote https://api.scrapesilo.com/mcp

(or pass --header "Authorization: Bearer $SCRAPESILO_API_KEY" to skip OAuth.)

Tools

All submission tools accept wait (default true). wait: false or the wait deadline returns full jobs in data.jobs, including completed siblings. Poll the same IDs; disconnecting does not cancel work. Each distinct URL has its own job; multi-URL submissions share a batchId. Tool content text uses the application envelope below.

Choose a tool by the result you need. Each tool below has a callable name and an example. scrape_markdown and docs are MCP-only; the other operations also have REST endpoints.

Goal	Tool
Read a page as Markdown	`scrape_markdown`
Extract fields from HTML	`scrape_html`
Read a JSON endpoint	`scrape_json`
Extract fields after JavaScript runs	`scrape_browser`
Describe the fields to extract	`scrape_ai`
Summarize or interpret a page	`analyse`
Discover URLs from sitemaps	`map`
Get Facebook ads from a Library search URL	`extract_facebook_ads`
Get Google ads by website/domain	`extract_google_ads`
Get YouTube ads by website/domain	`extract_google_ads` with `platform: "youtube"`
Look up documentation and examples	`docs`

`scrape_markdown`

For reading a page as Markdown, use scrape_markdown with just a URL. No actions or selectors are needed.

{ "url": "https://example.com/", "mode": "full", "engine": "html" }

Parameter	Default	Description
`url`	Required	One URL or an array of URLs.
`mode`	`"full"`	`full` converts the entire page body. `article` selects the main article with Readability.
`engine`	`"html"`	`html` fetches static HTML. `browser` renders JavaScript first.
`wait`	`true`	Set to `false` to return jobs immediately and poll for results.

Use full for documentation, listings and complete page content. Article mode can omit content or return empty text on non-article pages.

To read an article after JavaScript runs:

{ "url": "https://example.com/article", "mode": "article", "engine": "browser" }

HTML costs 1 credit per URL and accepts up to 50 URLs. Browser accepts up to 10 URLs at browser pricing. The tool does not automatically retry with another engine.

Completed results contain data.markdown alongside the usual URL, job ID and diagnostics. The tool uses the same authenticated job queue and result tools as other scrapes. It converts page content without summarizing it. For custom selectors, headers, sessions or browser wait options, use the engine-specific tools. The DSL article extractor is body@fitMd, not body@mdFit.

`scrape_html`

Extract named fields from static HTML with Actions DSL. Requires url and actions. The engine is fixed; omit engine. Costs 1 credit per URL, with up to 50 URLs per call.

{
  "url": "https://scrapesilo.com/fixtures/article",
  "actions": { "title": "h1", "cta": "a.sf-cta@href" }
}

Supports HTTP options and custom headers. See scrape parameters for sessions and proxy settings.

`scrape_json`

Read a JSON endpoint. Requires url; omit actions to return the whole document, or provide a flat map of output names to JSON paths. Omit engine. Costs 1 credit per URL, with up to 50 URLs per call.

{
  "url": "https://api.example.com/products",
  "actions": { "names": "items[*].name" }
}

Supports HTTP options and custom headers. See JSON extraction.

`scrape_browser`

Extract named fields after JavaScript runs. Requires url and actions; omit engine. Accepts up to 10 URLs at browser pricing.

{
  "url": "https://scrapesilo.com/fixtures/js-list",
  "actions": {
    "items": { "selector": ".sf-item", "many": true, "output": "text" }
  }
}

Supports browser rendering options, sessions and proxies. Custom headers are rejected. See scrape parameters for supported options.

`scrape_ai`

Describe the fields you want to extract from the supplied page. Requires url and query; accepts optional wait. Costs 50 credits per job.

{
  "url": "https://scrapesilo.com/fixtures/article",
  "query": "Extract the page heading as title and the call-to-action URL as link."
}

The tool chooses and tests extraction actions, then returns actual engine data. Each result also includes the tested recipe, model metadata and extractionStatus: "complete". It extracts only the supplied page, with no sitemap discovery.

Do not pass engine, actions, outputSchema, model choices or proxy options. See AI extraction and limits for the full contract.

`extract_facebook_ads`

Collect Facebook ads from an Ads Library search URL without selectors or actions. Uses the existing browser extractor and costs 5 credits per job. Default limit is 100 (maximum 1,000).

{ "url": "https://www.facebook.com/ads/library/?active_status=active&ad_type=all&country=ALL&q=wellaray.com&search_type=keyword_unordered", "limit": 100, "useProxy": "us", "wait": false }

Returns available copy, images, videos and carousel cards. Supports the same proxy/session options as the Facebook REST endpoint. Inspect complete and stopReason for partial collections; poll returned jobs with get_job_result without resubmitting.

`extract_google_ads`

One HTTP tool for Google ads across platforms. Supply a domain; no browser, selectors, or BigQuery required. Default limit is 100 (maximum 1,000).

{ "domain": "hyggear.com", "platform": "youtube", "adFormat": "video", "limit": 100, "wait": false }

platform accepts google_play, google_maps, google_search, youtube, or google_shopping; omit it for all platforms. adFormat independently accepts text, image, or video. Use google_search for Search ads; YouTube placement does not imply video format.

Search results use discovery previews only, with no extra creative requests. On other platforms or unfiltered searches, supplied previews are fetched over HTTP for video URLs and copy, or to resolve missing image/text creatives. Ads with usable inline image/text content need no extra request. Only the representative discovery preview is extracted, not all variations. Inspect discovery complete/stopReason and per-ad creativeStatus separately. No OCR, transcription, or media download.

Uses existing JSON jobs and HTTP transport options at 1 credit/job. Poll returned jobs with get_job_result. The former separate YouTube tool is replaced by platform: "youtube". See parameters and response semantics.

Response shapes

Every tool result keeps the SDK content array with a JSON text item. Parse that text as { ok: true, data } or { ok: false, error: { code, message, details? } }. Invocation failures also set SDK isError: true.

Tool / outcome	Successful `data`
Any scrape tool or `analyse`, completed	Result array
map completed	Map aggregate
any submission unfinished or `wait: false`	`{ jobs: ScrapeJob[] }`
get_job	`{ job: ScrapeJob }`
list_jobs	`{ jobs: ScrapeJob[] }`
get_job_result	Stored result or null
list_job_results	Ordered results with positional nulls
docs	Catalog, fixture example or document value

A retrieved failed job, a failed stored result or mixed per-URL outcomes still use ok: true. Inspect their operation errors. Missing/foreign jobs and unknown tool names use NOT_FOUND; invalid arguments use VALIDATION_ERROR. Unknown protocol methods and malformed JSON-RPC messages retain protocol errors separately. See Errors and response contracts for codes and rollout compatibility.

`scrape` (legacy)

Use the tools above for new integrations. This interface remains available for existing clients.

Same surface as POST /scrape. Args: { url, engine, actions?, sessionId?, options?, useProxy?, myProxyUrl?, myProxyConfig?, solveCaptcha? } (url may be an array). engine is "html", "json" or "browser". HTML/browser require actions; JSON accepts a flat path map or omission for the whole document.

When complete, data contains an array of { url, data, tookMs, jobId, batchId?, error? }. Every item has jobId. batchId is set only when more than one distinct URL is admitted. Failed URLs: data: null + error; siblings still return.

Deterministic: engine: "html" | "browser" + actions (see Actions DSL).
Prefer analyse for interpretation / long-form page reading.

json — whole API document

{ "url": "https://api.example.com/products", "engine": "json" }

Optional actions such as { "names": "items[*].name" } select fields. See JSON engine.

html — article fixture

{
  "url": "https://scrapesilo.com/fixtures/article",
  "engine": "html",
  "actions": {
    "title": "h1",
    "body": "p",
    "cta": "a.sf-cta@href"
  }
}

{
  "ok": true,
  "data": [
    {
      "url": "https://scrapesilo.com/fixtures/article",
      "data": {
        "title": "…",
        "body": "…",
        "cta": "https://scrapesilo.com/…"
      },
      "tookMs": 120,
      "jobId": "job_…"
    }
  ]
}

browser — JS-rendered list

Raw HTML is empty; items appear after JavaScript. Use engine: "browser".

{
  "url": "https://scrapesilo.com/fixtures/js-list",
  "engine": "browser",
  "actions": {
    "items": {
      "selector": ".sf-item",
      "many": true,
      "output": "text"
    }
  }
}

`analyse`

Document analysis over page markdown (not a CSS plan). Use for outlines, summaries, yes/no flags, compliance-style reading. Args: { url, query, headers?, sessionId?, options?, useProxy?, myProxyUrl?, myProxyConfig? }. When complete, data contains an array of { url, data: { result, evidence, confidence }, tookMs, jobId, batchId?, error? }.

{
  "url": "https://scrapesilo.com/fixtures/prose",
  "query": "Outline the article sections and state the main claim in one sentence."
}

`map`

Discover URLs from a website’s sitemaps, with optional domain and URL filters. Pages are never crawled. Args: { url, limit?, scope?, maxPathDepth?, include?, exclude?, wait? }. See POST /map for parameter ranges, defaults, scope examples, discovery limits, and migration from removed fields. wait defaults to true.

Completed data is { tookMs, sitemaps: [{ domain, url, links }], jobId?, batchId?, error? }; unfinished jobs return data: { jobs: [...] }. The aggregate concatenates successful per-seed sitemaps in order, sums known durations (null if none), and reports the first failure; check error for partial failure. Unmetered.

`docs`

Look up documentation and the noindex fixture examples. Not a REST route.

`topic`	Value under `data`
omitted	Catalog: doc slugs (with `.md` URLs) and fixture slugs
`quickstart`, `engines`, `actions-dsl`, `scrape`, `analyse`, `errors`, `mcp`, …	That page as markdown
`article`, `listing`, `nested`, `table`, `json`, `prose`, `js-list`, `form`	Canonical `scrape` / `analyse` args against `https://scrapesilo.com/fixtures/…`

{ "topic": "actions-dsl" }

{ "topic": "article" }

Call this before inventing selectors or engines.

`get_job`

args: { id: string } → data: { job: ScrapeJob }. Equivalent to GET /jobs/:id.

`list_jobs`

args: { status?, engine?, apiKeyId?, batchId?, legacyParentId?, limit? } → data: { jobs: ScrapeJob[] }. Equivalent to GET /jobs. Filter by batchId to pull every row from one multi-URL scrape.

`get_job_result`

args: { id: string } → data containing the stored result or JSON null. Equivalent to GET /jobs/:id/result. Null can mean pending or unavailable historical output; use get_job for status.

`list_job_results`

args: { batchId: string } → data containing all visible stored results in batch order, with positional nulls for unfinished or unavailable output. Use list_jobs for status. Missing or forbidden IDs/batches return a tool error.

Copy for an agent

MCP URL: https://api.scrapesilo.com/mcp. Auth: OAuth (Claude Code /mcp Authenticate) or Authorization: Bearer sf_…. Tools: scrape_markdown, scrape_html, scrape_json, scrape_browser, scrape_ai, extract_google_ads, extract_facebook_ads, legacy scrape, analyse, map, docs, get_job, list_jobs, get_job_result, list_job_results. Call docs with no topic for the catalog, or topic set to a slug (actions-dsl, article, …). scrape requires engine (html|json|browser); html/browser require actions, json allows omitted or empty actions. analyse requires url + query. Parse tool text as { ok, data } or { ok: false, error }; unfinished submissions return data.jobs. Poll those IDs without resubmitting.

{
  "mcpServers": {
    "scrapesilo": {
      "type": "http",
      "url": "https://api.scrapesilo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${SCRAPESILO_API_KEY}"
      }
    }
  }
}

For page reading, call scrape_markdown:

{ "url": "https://scrapesilo.com/fixtures/article" }

For named fields, call scrape_html:

{
  "url": "https://scrapesilo.com/fixtures/article",
  "actions": {
    "title": "h1",
    "body": "p",
    "cta": "a.sf-cta@href"
  }
}

analyse tool args:

{
  "url": "https://scrapesilo.com/fixtures/prose",
  "query": "Outline the article sections and state the main claim in one sentence."
}

MCP

Claude Code (OAuth — recommended)

API key (CI / headless)

Cursor / other clients

Tools

scrape_markdown

scrape_html

scrape_json

scrape_browser

scrape_ai

extract_facebook_ads

extract_google_ads