Engines

Choose /scrape/html, /scrape/json, /scrape/browser or /scrape/ai; omit engine from these bodies. Legacy /scrape still selects a deterministic engine through its engine field.

At a glance

Engine	JavaScript	Credits / URL	Use when
html	No	1	Static pages, server-rendered HTML, RSS and sitemaps.
json	No	1	JSON APIs: whole documents or selected fields.
browser	Yes	5	Pages that need JavaScript, interaction or login flows.
ai	As needed	50	Describe the fields to extract instead of writing actions.

HTML uses a plain fetch, JSON fetches and parses the response, and browser uses headless Chromium. AI selects from these engines; Jev decides whether HTML needs browser rendering. Typical HTML latency is about 100–400 ms and browser latency about 2–5 s. Actual timing depends on the upstream page; AI also depends on its extraction loop.

Cost is per URL — see Pricing & quotas for the full table and how multi-URL requests are charged.

For Markdown through MCP, use scrape_markdown with mode: "full" | "article" and engine: "html" | "browser". It supplies the extraction actions for you and uses the selected engine’s normal pricing and queue.

Decision tree

1. Try html first. Cheapest (1 credit), fastest. Static / SSR pages usually have the fields in the raw HTML.
2. Escalate to browser when html returns null or empty fields, the page is JS-rendered, or you need click / fill / wait / navigation.
3. Use analyse (POST /analyse) when you want interpretation — outline, summary, “is X present?” — not exact CSS fields. Analyse is one LLM pass over page markdown; it does not emit CSS actions.

Prove the split on the fixture pages: /fixtures/article is static (html works); /fixtures/js-list is empty until JS runs.

AI orchestration

AI accepts one URL and a natural-language query. It fetches the page, uses Jev to decide whether browser rendering is needed, then uses cleaned HTML with relevant regions retained in full and irrelevant regions reduced to stubs. JSON responses use the JSON engine and are shown whole. The model tests read-only actions within five model steps, including submission. Successful submitted actions select the actual engine data and reusable recipe. Only the provided page is extracted. No output schema, recipe persistence or caller browser/session/proxy settings are accepted. See the AI request and limits.

Options matrix

This matrix describes HTML/browser runtime behavior, including the legacy /scrape interface. The precise /scrape/html endpoint rejects browser-only options instead of ignoring them. scrape_markdown exposes only URL, mode, engine and wait; use the engine-specific tools for these advanced options.

This matrix describes the legacy /scrape behavior. The precise HTML/JSON routes reject browser-only fields rather than ignoring them.

Option	html	browser
options.waitFor	Ignored	Lifecycle (load / domcontentloaded / networkidle / commit), ms delay, or CSS selector. Default goto lifecycle when waitFor is ms/selector: domcontentloaded.
options.timeoutMs	Honoured — AbortController on the fetch. Default 30s, max 120s.	Honoured — an API-side deadline for page setup, navigation, waits and actions. Default 30s, max 120s.
resolution	Ignored	Viewport: "desktop" (1280×800), "mobile" (390×844 + mobile UA), or { width, height }.
headless	Ignored	Default true. false renders into a real display — better fingerprint stealth, higher CPU/RAM.
blockAds	Ignored	Default true. false stops refusing known ad/tracker hosts.
solveCaptcha	Ignored	Opt-in 2captcha on detected reCAPTCHA; outcome on result.captcha.
sessionId	Matching cookies only (Cookie header). localStorage ignored.	Full storageState: cookies + per-origin localStorage.

Worked examples

HTML example

Static article. Raw HTML already has h1, p, and a.sf-cta.

{
  "url": "https://scrapesilo.com/fixtures/article",
  "engine": "html",
  "actions": {
    "title": "h1",
    "body": "p",
    "cta": "a.sf-cta@href"
  }
}

Browser example

Form needs interaction: fill the query, click submit, then extract the results.

{
  "url": "https://scrapesilo.com/fixtures/form",
  "engine": "browser",
  "actions": {
    "type": { "fn": "fill", "selector": "#sf-q", "args": "widget" },
    "submit": { "fn": "click", "selector": "#sf-submit" },
    "title": ".sf-result-title",
    "count": ".sf-result-count"
  }
}

Why HTML is empty

/fixtures/js-list has no .sf-item nodes in the raw HTML — they appear after client JS. An html scrape returns empty items; browser does not.

{
  "url": "https://scrapesilo.com/fixtures/js-list",
  "engine": "html",
  "actions": {
    "items": { "selector": ".sf-item", "many": true, "output": "text" }
  }
}

{
  "url": "https://scrapesilo.com/fixtures/js-list",
  "engine": "browser",
  "actions": {
    "items": { "selector": ".sf-item", "many": true, "output": "text" }
  }
}

html — plain fetch

html fetches the page with undici (fetch, follow redirects) and runs the action tree over the static DOM. No JS execution.

fn actions wait, goto, click, fill, selectOption, and 2fa are rejected (warning: use browser). evaluate is allowed as sync JSDOM eval only — no async / Promise. Full fn matrix: Actions DSL.

Non-2xx

A non-2xx response is a failure for html: that URL’s result has data: null, error set, and the captured HTTP status. Sibling URLs in the same request still return. Browser does not fail the run on a non-2xx page — it may still run actions and reports the real page status.

json — direct JSON

Fetch a JSON API without DOM parsing or a browser. Omit actions (or use {}) to return the entire document, including arrays, strings, numbers, booleans and null. With actions, each output name maps to a JSON path:

{
  "url": "https://api.example.com/products",
  "engine": "json",
  "actions": { "first": "items[0].name", "names": "items[*].name", "all": "" }
}

Simple dot/bracket paths return one value. JSONPath wildcard/recursive paths return matches; missing paths return null. Empty paths select the whole document. Paths only read own document properties. JSONPath filters/scripts requiring expression evaluation are disabled and fail; no user code executes.

Valid JSON is accepted regardless of Content-Type. Embedded HTML and entities are preserved verbatim. Malformed JSON and non-2xx responses fail the job; successful JSON null is distinguished by the absence of error. Headers, proxies, captured cookies and timeoutMs work as on HTML. Browser/DOM actions, nested action groups, waitFor, resolution, headless, blockAds and solveCaptcha are rejected. localStorage is unavailable.

JSON and HTML share the plain-HTTP queue concurrency budget. Both cost 1 credit per URL and allow 50 URLs per request. Existing HTML body@json extraction remains supported.

browser — real Chromium

Full browser runtime: navigation, click / fill / select, evaluate (with async / timeout race), 2FA via otpauth. Pages run in Chromium with stealth and ad-blocking on by default.

options.waitFor controls when navigation is considered done — a lifecycle keyword (networkidle is safest for modern SPAs; domcontentloaded is fastest), a number of ms, or a CSS selector to wait for. Ignored on html. options.timeoutMs caps the per-page run at up to 120 s on all engines. A timed-out browser run releases its remote session before the job is completed as failed; cleanup has a separate maximum of 5 s.

Authenticated sessions

Capture a logged-in browser session once via the dashboard’s live recorder, then replay it by passing the captured id as top-level sessionId. Browser restores cookies + per-origin localStorage. HTML/JSON send only the matching cookies on the initial request (localStorage is unreachable from a plain HTTP fetch). List saved sessions with GET /sessions.

Multiple URLs per request

url accepts either a single URL or an array. The same actions run against every URL; each distinct URL is an independently scheduled queue job and returns one result item. Operation failures do not overwrite sibling results. Browser jobs each use an isolated context initialized from their saved profile; sibling mutations are not shared. Each URL is charged independently (cost = per-engine credit × URL count).

Engine	Max URLs per request
html	50
json	50
browser	10

Current user-plan limits count running URL jobs across requests, keys and API replicas. Resource caps also apply; the engine no longer starts a separate per-request URL pool. HTTP 202 returns jobs for polling when requested asynchronously or the wait deadline expires.

{
  "url": [
    "https://scrapesilo.com/fixtures/article",
    "https://scrapesilo.com/fixtures/listing"
  ],
  "engine": "html",
  "actions": { "title": "h1" }
}

Proxies

Any scrape accepts these optional top-level fields:

Field	What it does
useProxy: true	Route the request through the built-in residential pool, when no BYO fields are set.
useProxy: "us"	Same, but an ISO 3166-1 alpha-2 country code geo-targets the built-in pool. Ignored when BYO is set.
myProxyUrl	BYO proxy as a single URL with embedded creds, e.g. http://user:pass@host:port.
myProxyConfig	BYO proxy as { server, username?, password? }.

The proxy applies to all scrape engines.

{
  "url": "https://scrapesilo.com/fixtures/article",
  "engine": "html",
  "actions": { "title": "h1" },
  "useProxy": "de"
}

Copy for an agent

ScrapeSilo engines: html (1 credit, plain fetch), json (1 credit, direct JSON), browser (5 credits, Chromium), ai (50 credits, query-driven extraction).
Try html first. Escalate to browser when fields are empty, the page needs JS, or you need click/fill/wait. Use POST /analyse for interpretation, not scrape.

Rules:
- Prefer POST /scrape/html, /scrape/json, /scrape/browser; omit engine. POST /scrape/ai takes one url and query.
- Legacy POST /scrape: engine is "html" | "json" | "browser". actions required for html/browser; optional flat path map for json. No query on /scrape.
- Specific HTML/JSON routes reject browser options. Legacy html ignores options.waitFor. Deterministic timeoutMs defaults to 30s, max 120s. AI controls its own budgets.
- resolution, headless, blockAds, solveCaptcha are browser-only.
- sessionId: html/json = cookies only; browser = cookies + localStorage.
- html rejects fn wait/goto/click/fill/selectOption/2fa. evaluate is sync JSDOM only.
- html non-2xx → data null + error + status. Caps: html/json 50 URLs, browser 10.

curl -sS -X POST https://api.scrapesilo.com/scrape \
  -H "Authorization: Bearer $SCRAPESILO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "url": "https://scrapesilo.com/fixtures/article",
        "engine": "html",
        "actions": { "title": "h1", "body": "p", "cta": "a.sf-cta@href" }
      }'

curl -sS -X POST https://api.scrapesilo.com/scrape \
  -H "Authorization: Bearer $SCRAPESILO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "url": "https://scrapesilo.com/fixtures/form",
        "engine": "browser",
        "actions": {
          "type": { "fn": "fill", "selector": "#sf-q", "args": "widget" },
          "submit": { "fn": "click", "selector": "#sf-submit" },
          "title": ".sf-result-title",
          "count": ".sf-result-count"
        }
      }'