Scrape endpoints

For supported sites, specialty extractors provide dedicated routes without selectors. They reuse the existing jobs and result polling.

Prefer the engine-specific endpoints below. They select the engine through the URL, so omit engine from the body. Unknown fields and unsupported engine options are rejected.

Endpoint	MCP tool	Input	Credits
POST /scrape/html	scrape_html	URL(s), required actions, HTTP options	1/URL
POST /scrape/json	scrape_json	URL(s), optional flat JSON-path actions, HTTP options	1/URL
POST /scrape/browser	scrape_browser	URL(s), required actions, browser options; no custom headers	5/URL
POST /scrape/ai	scrape_ai	One URL and query	50/job

For page reading over MCP, scrape_markdown accepts url, optional mode (full or article), engine (html or browser) and wait. No actions or selectors are required. This is an MCP convenience tool; there is no /scrape/markdown HTTP endpoint. HTTP clients use /scrape/html or /scrape/browser with actions: { "markdown": "body@md" } (or body@fitMd for article detection).

HTML/JSON accept at most 50 URLs; browser accepts 10. HTML/JSON support headers, session/proxy fields and options.timeoutMs, but reject rendering options and solveCaptcha. Browser supports rendering, session/proxy fields and solveCaptcha but rejects headers. AI accepts only url and query.

// POST /scrape/html
{ "url": "https://example.com/", "actions": { "title": "h1" } }

// POST /scrape/json
{ "url": "https://example.com/data.json", "actions": { "items": "items[*]" } }

// POST /scrape/browser
{ "url": "https://example.com/", "actions": { "title": "h1" }, "options": { "waitFor": "h1" } }

AI extraction

// POST /scrape/ai
{ "url": "https://example.com/", "query": "Extract the page heading as title" }

AI fetches the supplied page, uses the JSON engine for JSON responses, or cleans the HTML and uses Jev to decide whether rendering is needed and which parts of the page are relevant to the query. The model sees the relevant parts in full; JSON is shown whole. The extraction model generates read-only actions, tests them through the existing engine and submits successful actions. The server returns the actual tested engine data without running the final actions again. Query length is 1–4,000 characters. No caller model selection, output schema, sessions, headers or proxy settings are accepted.

There are at most five model steps including final submission and five extraction tests. One Jev decision (skipped for JSON responses), optional browser preparation and, for rendered pages, a relevance-only Jev decision precede the model loop. Capability timeouts are 15 seconds; the pipeline cancellation budget is 90 seconds plus in-flight engine cleanup. Relevant page parts are never truncated and there is no size cap; actions always run on the original page. Only the supplied page is extracted; discovery and multi-page aggregation are not part of this operation.

Generated actions use the existing DSL’s read-only subset: selectors, attributes, repeated rows, nested outputs and embedded JSON/path reads. Arbitrary code, function actions, clicking/typing, CAPTCHA solving and screenshots are unavailable. Static HTML/JSON tests reuse the original fetched response. Browser preparation and tests navigate through existing engine invocations.

A completed response is a result array containing canonical engine metadata, data, a tested recipe: { url, engine, actions }, extractionStatus: "complete", and optional model/steps/usage. data is only the extracted JSON; numeric status remains HTTP status. Extracted keys named recipe, status or data are preserved. No model-authored final data or evidence IDs are accepted. Recipes are tested on this page only, without future guarantees or managed storage. Errors use data: null plus error; no successful tested submission within the budget is a failure. Token usage describes the extraction model, excluding Jev.

AI reserves 50 credits once at admission. Internal calls do not create or charge additional jobs. AI obeys user concurrency, its own queue cap (default 2), and shares browser capacity for the whole job.

Legacy compatibility

POST /scrape and MCP scrape remain available for existing deterministic clients. Their existing engine-tagged contract is described below; they do not accept engine: "ai".

The legacy endpoint. Validates the body and queues one durable ScrapeJob per distinct URL for the html, json or browser engine. Completed synchronous calls return an array of stored results.

engine is 'html' | 'json' | 'browser'. HTML/browser require actions; JSON allows omitted actions for the whole document. Interpretation of a page belongs on POST /analyse, not here.

Queue and asynchronous calls

Every distinct URL creates a durable job; multiple jobs share a batchId. By default, every scrape endpoint (including /scrape/ai and legacy /scrape) waits for all submitted jobs and returns its extracted results with HTTP 200.

Use ?wait=false on any scrape endpoint (for example POST /scrape/ai?wait=false) to receive HTTP 202 with job rows immediately. Poll GET /jobs/:id using each returned row’s id; its result is populated when complete. A synchronous request that reaches the server’s wait deadline also returns HTTP 202 with job rows. Do not submit another scrape to poll.

The response includes X-Scrape-Job-Ids (comma-separated job IDs) and X-Scrape-Batch-Id when applicable. Closing the connection stops waiting but leaves the job running. Pending jobs survive server restarts; interrupted running jobs are failed rather than automatically replaying potentially mutating actions.

Request

POST /scrape
Authorization: Bearer <token>
Content-Type:  application/json

Body (any field marked optional may be omitted):

{
  url: string | string[];   // one http(s) URL, or an array (see Multiple URLs)
  engine: 'html' | 'json' | 'browser';
  actions?: BrowserActions | Record<string, string>; // required for html/browser; JSON uses flat paths

  sessionId?: string;       // replay a recorded auth session — /docs/engines#authenticated-sessions
  options?: {
    waitFor?:    'load' | 'domcontentloaded' | 'networkidle' | 'commit' | number | string;
    timeoutMs?:  number;    // capped at 120000
    resolution?: 'desktop' | 'mobile' | { width: number; height: number }; // browser only
    headless?:   boolean;   // browser only, default true
    blockAds?:   boolean;   // browser only, default true
  };

  useProxy?:      boolean | string;  // true = built-in pool; "us" geo-targets it
  myProxyUrl?:    string;            // BYO: "http://user:pass@host:port"
  myProxyConfig?: { server: string; username?: string; password?: string };
  solveCaptcha?:  boolean;           // browser only — opt-in 2captcha on detected reCAPTCHA
}

The server rejects HTML/browser requests where actions is missing. JSON accepts omitted or empty actions and returns the whole parsed document. solveCaptcha is ignored on html (no JS runtime to inject a token). Outcome of a solve, when attempted, is captcha on that URL’s result.

Proxies

With useProxy: true and no BYO fields, the request uses the built-in residential pool. A 2-letter ISO country code ("us", "de") geo-targets that pool. Country is ignored when myProxyUrl / myProxyConfig is set. See Engines → Proxies.

Multiple URLs

url may be an array. The same actions run against each URL as an independently scheduled job. Exact duplicate URL strings are removed, preserving first-occurrence order. Browser jobs each use an isolated context initialized from the saved profile; sibling context mutations are not shared. A failed operation does not overwrite its siblings. Invalid or blocked input URLs reject admission before any jobs are inserted. Each URL is charged separately. Per-request caps: html 50, json 50, browser 10. See Engines → Multiple URLs.

Response

HTTP 200 returns an array — one item per deduplicated URL in input order. HTTP 202 returns full job rows instead:

Array<{
  url: string;                          // final URL, after redirects
  status?: number;                      // page HTTP status; absent if no response
  data: Record<string, unknown> | unknown[] | string | number | boolean | null; // JSON whole-document values; failures have error
  tookMs: number;
  antibot: Array<{                      // empty when the page is clean
    provider: string;
    detection: string;
    blocked: boolean;
    version?: string;
    sitekey?: string;
  }>;
  captcha?: {                           // only if a solve was attempted
    provider: string;
    solved: boolean;
    tookMs: number;
    error?: string;
  };
  error?: { message: string; name: string; stack: string | null; cause: unknown };
  jobId: string;                  // always present — row in /jobs
  batchId?: string;                     // only when the request had more than one URL
}>

Ids

Every item includes jobId. Inspect the full row later via the Jobs API. batchId is set only when more than one distinct URL was admitted.

Errors

HTTP-level failures (the whole request is rejected):

Status	Code	Cause
400	VALIDATION_ERROR / MALFORMED_REQUEST	Invalid fields or malformed body.
400	BLOCKED_URL	Admission rejected a URL.
401	UNAUTHENTICATED	Bearer credential did not resolve.
402	QUOTA_EXCEEDED	Credit admission failed; credit values are under error.details.
503	CAPACITY_EXCEEDED	Pending queue is full; Retry-After: 5.

Failures use { error: { code, message, details? } }; see Errors and response contracts. Accepted operation failures remain on individual jobs/results: HTTP 200 can contain failed URLs with data: null and error, while siblings still return.

Example

curl -sS -X POST https://api.scrapesilo.com/scrape \
  -H "Authorization: Bearer sf_…" \
  -H "Content-Type: application/json" \
  -d '{
        "url": "https://scrapesilo.com/fixtures/article",
        "engine": "html",
        "actions": { "title": "h1", "body": "p", "cta": "a.sf-cta@href" }
      }'

[
  {
    "url": "https://scrapesilo.com/fixtures/article",
    "status": 200,
    "data": {
      "title": "How to pin an article field",
      "body": "This lead is the body field. Select the first paragraph after the heading.",
      "cta": "https://scrapesilo.com/docs/quickstart"
    },
    "tookMs": 287,
    "antibot": [],
    "jobId": "job_…"
  }
]

The same body is the MCP scrape tool args. See MCP.

Copy for an agent

Use this as a self-contained prompt. engine is "html", "json" or "browser". HTML/browser require a CSS actions tree; JSON accepts an optional flat output-name → path map (see Actions DSL). Do not send a planner prompt on this endpoint. A completed HTTP 200 response is a result array; every item has jobId. HTTP 202 returns jobs to poll without resubmitting.

curl -sS -X POST https://api.scrapesilo.com/scrape \
  -H "Authorization: Bearer sf_…" \
  -H "Content-Type: application/json" \
  -d '{
        "url": "https://scrapesilo.com/fixtures/article",
        "engine": "html",
        "actions": { "title": "h1", "body": "p", "cta": "a.sf-cta@href" }
      }'

JSON and plain-text endpoints

Use the json engine for JSON APIs; see JSON engine. The older html extraction below remains supported for compatibility. Responses with application/json, a +json media type, or text/plain are treated as literal body text, so embedded markup and entities are preserved. HTML responses retain normal DOM extraction. Missing/other content types retain the existing HTML interpretation.

{
  "url": "https://cloudflare-dns.com/dns-query?name=example.com&type=MX",
  "engine": "html",
  "headers": { "Accept": "application/dns-json" },
  "actions": { "answer": "body@json" }
}

body@json:path.to.field selects part of the parsed JSON. body@text trims surrounding whitespace; body@textOriginal preserves it. Existing extraction options still apply. No new engine or queue is needed.

Request headers

headers is an optional string-to-string object for the HTML/JSON engines. Browser requests with headers are rejected. Header names are case-insensitive; duplicate names with different casing, invalid characters, and transport-managed fields such as Host, Content-Length, Accept-Encoding, and Proxy-* are rejected. Limits are 64 headers and 8192 characters per value.

Custom headers apply to each input URL and same-origin redirects. After a cross-origin redirect, all supplied headers are dropped for the rest of that redirect chain. Redirect targets pass the public-URL guard. An explicit Cookie header overrides stored-session cookies. These are target-site headers, not credentials for calling ScrapeSilo.

Headers and custom proxy inputs are preserved in the stored request and omitted when serializing public job reads. A copied historical request requires those inputs to be re-entered. They are not copied into response diagnostics.

Response diagnostics

When an upstream response is reached, each result includes response with requestedUrl, finalUrl, status, statusText, and contentType. These details survive later extraction/wait failures and are stored with the job result, including failed queued scrapes.

For HTML-engine non-2xx responses, response.bodyPreview contains at most 2048 characters and bodyTruncated indicates that more data was present or the read was stopped. Response cookies are not included. A preview is untrusted upstream content and can contain reflected data. Browser diagnostics describe the initial navigation response, not later action-triggered navigations; browser HTTP error statuses retain their existing behavior.

If no response arrives, diagnostics are absent. After a redirect-chain failure they describe the last received response. This is response-level evidence, not a raw wire capture.