POST /map

Discover URLs from a website’s sitemaps, with optional domain and URL filters. Pages are never crawled. Use /map to build a list of URLs, then feed them to /scrape for extraction.

Request

POST /map
Authorization: Bearer <token>
Content-Type:  application/json

{
  url: string | string[];
  limit?: number;
  scope?: "domain" | "host" | "any";
  maxPathDepth?: number;
  include?: string;
  exclude?: string;
}

Parameter	Type and range	Default	Meaning
url	URL string or array of 1–25 URL strings	Required	Seed URL or URLs. Each distinct seed creates one job. Accepts a page URL or a direct .xml sitemap URL.
limit	Integer, 1–5,000	5000	Maximum unique matching URLs returned per seed. Filtering happens before the limit.
scope	"domain", "host", or "any"	"domain"	Hostname restriction for returned URLs; see the examples below. Does not change which sitemaps are fetched.
maxPathDepth	Integer, 0–10	No path-depth restriction	Maximum number of nonempty path segments in a returned URL. / has 0 segments; /products/shoes has 2. Query strings and fragments do not count.
include	String, at most 300 characters	No inclusion filter	Return only URLs matching this case-sensitive JavaScript regex, tested against the full URL. Invalid or unsafe expressions are rejected.
exclude	String, at most 300 characters	No exclusion filter	Omit URLs matching this case-sensitive JavaScript regex, tested against the full URL. Invalid or unsafe expressions are rejected. Exclusion wins over inclusion.

Scope examples

domain removes one leading www. from the seed hostname and allows that hostname and its subdomains. host allows only the seed’s exact hostname. any applies no hostname restriction to returned URLs.

Seed hostname	Scope	Allowed hostnames in results
www.example.com	domain (default)	example.com, www.example.com, shop.example.com, and other subdomains of example.com
shop.example.com	domain (default)	shop.example.com and descendants such as cdn.shop.example.com; excludes example.com and blog.example.com
www.example.com	host	Only www.example.com
www.example.com	any	Any hostname listed in the fetched sitemaps

All other URL filters still apply with every scope.

Discovery and limits

Map reads a direct .xml seed, otherwise it reads sitemap declarations from robots.txt, falling back to /sitemap.xml when no eligible declarations exist. It follows nested sitemaps breadth-first. Only sitemaps on the seed’s site are fetched: the seed hostname with one leading www. removed, and its descendants. This fetch boundary is independent of scope; even scope: "any" does not fetch external sitemaps. Pages are never fetched. Every fetch and redirect is checked against the private-network policy.

Per seed, traversal stops when it reaches limit unique matching URLs, 3 sitemap levels (the first sitemap is level 1), 500 sitemaps, 600 requests (including robots.txt and redirects), 50 MiB of response data, or 2 minutes. It returns the links found so far. A sitemap that fails to load is skipped.

Each distinct seed creates one queued job. Sitemap URLs discovered inside that seed’s traversal do not create extra jobs. Map does not consume credits, but shares the user’s running-job concurrency allowance with scrape and analyse.

Response and job handling

{
  tookMs: number | null;
  jobId?: string;   // singleton
  batchId?: string; // multiple distinct seeds
  error?: { message: string; /* … */ };
  sitemaps: Array<{
    domain: string;
    url:    string | string[] | null;  // visited sitemap URL(s), null if no loc entries were found
    links:  string[];
  }>;
}

HTTP 200 returns this aggregate after all jobs finish. Sitemaps are concatenated in seed order; failed seeds contribute no sitemaps and the first error is included. Check error even when usable sitemaps are present. tookMs sums known per-seed durations, or is null if none are known; it is not HTTP wait time.

?wait=false or the wait deadline returns HTTP 202 with full job rows, never a partial aggregate. Poll the same IDs or batch without resubmitting. Submission headers include X-Scrape-Job-Ids and, for batches, X-Scrape-Batch-Id. Disconnect does not cancel jobs. Result retrieval returns individual per-seed values.

Errors

Invocation failures use structured errors, including VALIDATION_ERROR, BLOCKED_URL, UNAUTHENTICATED and CAPACITY_EXCEEDED. An accepted seed failure remains an operation outcome in the aggregate error, even with HTTP 200. Map does not incur a credit-quota rejection.

Unknown request fields are rejected. The old depth, allowSubdomains, and ignoreSitemap fields are no longer accepted:

• Replace depth with maxPathDepth to filter URL path depth. It does not control sitemap nesting.
• Replace allowSubdomains: false with scope: "host", or allowSubdomains: true with scope: "any" to preserve those explicit hostname-filter settings. The new default is scope: "domain".
• Remove ignoreSitemap. Sitemaps are the only discovery mechanism.

Examples

curl -sS -X POST https://api.scrapesilo.com/map \
  -H "Authorization: Bearer sf_…" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com", "limit": 100, "scope": "domain", "maxPathDepth": 2, "include": "/blog/" }'

{
  "tookMs": 412,
  "sitemaps": [
    {
      "domain": "example.com",
      "url": "https://example.com/sitemap.xml",
      "links": ["https://example.com/blog/a", "https://example.com/blog/b"]
    }
  ]
}

/map is also exposed as the map MCP tool.