Jobs API

Every distinct submitted URL creates one ScrapeJob. Exact duplicate URL strings are removed in first-occurrence order. Multiple jobs share a `batchId`; there is no parent job. Jobs complete independently. Deterministic scrape, AI extraction, analyse and map use the same durable queue. AI admits one seed and one extraction target within a single job.

Identifiers (`job_…`, `batch_…`, `ak_…`) are opaque strings: a prefix and 32 hexadecimal characters. Records created before this format keep their original, differently shaped IDs, so store and compare IDs as whole strings and never parse them.

API-key authentication is sufficient without a user session and only exposes that key's jobs. Clerk users can read jobs across their own keys. Foreign IDs and batches return `404 NOT_FOUND` using the [structured error contract](/docs/api/errors).

## Submission and polling

`POST /scrape`, `/scrape/html`, `/scrape/json`, `/scrape/browser`, `/scrape/ai`, `/analyse` and `/map` accept `?wait=true|false` (default true). HTTP 200 contains completed result arrays for scrape/analyse, or a map aggregate. HTTP 202 contains full jobs in submission order when `wait=false` or the wait deadline expires, including siblings that already completed. Responses include `X-Scrape-Job-Ids` and, for batches, `X-Scrape-Batch-Id`.

Poll the existing IDs or batch; never submit the request again just to wait. Disconnecting detaches the waiter without cancelling jobs. Expired leases fail individually; potentially mutating actions are never automatically replayed. There is no cancellation or retry endpoint.

## `GET /jobs`

```text
GET /jobs?status=failed&engine=html&limit=20
Authorization: Bearer <token>
```

| Filter | Meaning |
|---|---|
| `status` | pending, running, completed, failed, cancelled (historical only) |
| `engine` | html, json, browser, ai, analyse, map |
| `limit` | 1–200; default 50; newest first |
| `batchId` | Complete visible batch in submission order, ignoring history limit |
| `legacyParentId` | Jobs migrated from an old request-level queue parent |
| `apiKeyId` | Clerk history filter for an owned key; API-key callers remain scoped to their own key |

Returns full public ScrapeJob rows:

```ts
Array<{
  id: string;
  apiKeyId: string;
  userId: string | null;
  engine: 'html' | 'json' | 'browser' | 'ai' | 'analyse' | 'map';
  status: 'pending' | 'running' | 'completed' | 'failed' | 'cancelled';
  batchId: string | null;
  batchPosition: number;
  legacyParentId: string | null;
  request: object; // operation input, scalar url, engine; private inputs omitted
  result: object | null; // ScrapeResult | AiScrapeResult | AnalyseResult | MapResult
  error: object | null;
  recordingUrl: string | null;
  replayUrl: string | null;
  createdAt: string;
  startedAt: string | null;
  completedAt: string | null;
}>
```

Public requests omit custom headers and custom proxy inputs. Private credentials and lease fields are not returned. Copying a historical request requires those inputs to be re-entered. Extracted data and normal actions/query/URL remain user content; this is not a universal secret-redaction system.

## `GET /jobs/:id`

Returns one full job or 404. The dashboard's [/jobs](/jobs) page uses these job APIs.

```bash
curl -sS https://api.scrapesilo.com/jobs/job_example \
  -H 'Authorization: Bearer sf_…'
```

## Results only

| Endpoint | Response |
|---|---|
| `GET /jobs/:id/result` | 200 stored result when terminal, including JSON null if historical output is unavailable; 202 JSON null while pending/running |
| `GET /jobs/results?batchId=...` | Ordered result-or-null array; 200 when all visible jobs are terminal, 202 while any are unfinished |

Missing or foreign IDs/batches return 404. Positional nulls preserve the batch order. Read full jobs for status/error details. Map batch results are per-seed values; the synchronous map endpoint separately combines them into its aggregate.

MCP equivalents are `get_job`, `list_jobs`, `get_job_result` and `list_job_results`. MCP wraps successful job reads in `data: { job }` or `data: { jobs }`, and result-only reads in `data: resultOrNull` or `data: results`. MCP result-only nulls have no HTTP status; inspect full jobs to distinguish unfinished work from unavailable history.

## Limits and history

Current user-plan concurrency limits count running URL jobs across all requests, API keys and API replicas, including map. Plan changes affect the next claim. HTML/browser/analyse resource limits and map traversal capacity also apply. Metered credits and all submitted jobs are admitted in one transaction; map is unmetered. Ownerless service keys bypass user quotas and user concurrency, while resource/backlog caps still apply. Browser jobs use separate contexts initialized from their saved profile, without sharing mutations across sibling URLs.

Terminal history is retained for seven days; monthly usage counters remain authoritative for billing. Existing `ex_` IDs survive migration and resolve through `/jobs/:id`. Old parent UUIDs can be found through `legacyParentId`. Historical multi-seed maps retain one terminal row with `request.legacyAggregate: true`; its original aggregate output cannot be reconstructed as per-seed jobs. Historical batch ordering uses deterministic timestamp/ID order when the original order was never recorded.

REST `/executions` and MCP execution tools have been replaced. Dashboard `/executions` links redirect to `/jobs`; this does not restore the old API contract.

## AI results

Jobs can have `engine: "ai"` and request `{ engine: "ai", url, query }`. The stored result retains canonical engine metadata and extracted JSON only in `data`, with tested `recipe`, `extractionStatus: "complete"` and optional model/step/usage metadata alongside it. Numeric `status` retains HTTP meaning. Result reads return it unchanged. The final data is selected from successful executed actions, not written by the model. Failures preserve `data: null` plus `error`. Owner scoping and pending-result behavior match other engines.

Markdown agents.txt

Every distinct submitted URL creates one ScrapeJob. Exact duplicate URL strings are removed in first-occurrence order. Multiple jobs share a batchId; there is no parent job. Jobs complete independently. Deterministic scrape, AI extraction, analyse and map use the same durable queue. AI admits one seed and one extraction target within a single job.

Identifiers (job_…, batch_…, ak_…) are opaque strings: a prefix and 32 hexadecimal characters. Records created before this format keep their original, differently shaped IDs, so store and compare IDs as whole strings and never parse them.

API-key authentication is sufficient without a user session and only exposes that key’s jobs. Clerk users can read jobs across their own keys. Foreign IDs and batches return 404 NOT_FOUND using the structured error contract.

Submission and polling

POST /scrape, /scrape/html, /scrape/json, /scrape/browser, /scrape/ai, /analyse and /map accept ?wait=true|false (default true). HTTP 200 contains completed result arrays for scrape/analyse, or a map aggregate. HTTP 202 contains full jobs in submission order when wait=false or the wait deadline expires, including siblings that already completed. Responses include X-Scrape-Job-Ids and, for batches, X-Scrape-Batch-Id.

`GET /jobs`

GET /jobs?status=failed&engine=html&limit=20
Authorization: Bearer <token>

Filter	Meaning
`status`	pending, running, completed, failed, cancelled (historical only)
`engine`	html, json, browser, ai, analyse, map
`limit`	1–200; default 50; newest first
`batchId`	Complete visible batch in submission order, ignoring history limit
`legacyParentId`	Jobs migrated from an old request-level queue parent
`apiKeyId`	Clerk history filter for an owned key; API-key callers remain scoped to their own key

Returns full public ScrapeJob rows:

Array<{
  id: string;
  apiKeyId: string;
  userId: string | null;
  engine: 'html' | 'json' | 'browser' | 'ai' | 'analyse' | 'map';
  status: 'pending' | 'running' | 'completed' | 'failed' | 'cancelled';
  batchId: string | null;
  batchPosition: number;
  legacyParentId: string | null;
  request: object; // operation input, scalar url, engine; private inputs omitted
  result: object | null; // ScrapeResult | AiScrapeResult | AnalyseResult | MapResult
  error: object | null;
  recordingUrl: string | null;
  replayUrl: string | null;
  createdAt: string;
  startedAt: string | null;
  completedAt: string | null;
}>

`GET /jobs/:id`

Returns one full job or 404. The dashboard’s /jobs page uses these job APIs.

curl -sS https://api.scrapesilo.com/jobs/job_example \
  -H 'Authorization: Bearer sf_…'

Results only

Endpoint	Response
`GET /jobs/:id/result`	200 stored result when terminal, including JSON null if historical output is unavailable; 202 JSON null while pending/running
`GET /jobs/results?batchId=...`	Ordered result-or-null array; 200 when all visible jobs are terminal, 202 while any are unfinished

MCP equivalents are get_job, list_jobs, get_job_result and list_job_results. MCP wraps successful job reads in data: { job } or data: { jobs }, and result-only reads in data: resultOrNull or data: results. MCP result-only nulls have no HTTP status; inspect full jobs to distinguish unfinished work from unavailable history.

Limits and history

Terminal history is retained for seven days; monthly usage counters remain authoritative for billing. Existing ex_ IDs survive migration and resolve through /jobs/:id. Old parent UUIDs can be found through legacyParentId. Historical multi-seed maps retain one terminal row with request.legacyAggregate: true; its original aggregate output cannot be reconstructed as per-seed jobs. Historical batch ordering uses deterministic timestamp/ID order when the original order was never recorded.

REST /executions and MCP execution tools have been replaced. Dashboard /executions links redirect to /jobs; this does not restore the old API contract.

AI results

Jobs can have engine: "ai" and request { engine: "ai", url, query }. The stored result retains canonical engine metadata and extracted JSON only in data, with tested recipe, extractionStatus: "complete" and optional model/step/usage metadata alongside it. Numeric status retains HTTP meaning. Result reads return it unchanged. The final data is selected from successful executed actions, not written by the model. Failures preserve data: null plus error. Owner scoping and pending-result behavior match other engines.