Scrapers
ScrapeRouter supports multiple scraper backends with different capabilities. This guide explains how to choose a scraper, use common request/response fields, and handle scraper-specific options.
For the current list of available scrapers with descriptions and capabilities, see Scrapers.
Selecting a scraper
Set the scraper field on your request to either a scraper identifier or "auto":
-
auto— The system selects a scraper based on the target URL and options (e.g. whether JavaScript is required). If a scraper fails, the system automatically tries the next more advanced scraper. -
Scraper identifier — Use a specific scraper, e.g.
scraperouter/curl_cffi:0.13orscraperouter/playwright:1.55. Valid identifiers are listed on the Scrapers page.
JavaScript vs non-JavaScript scrapers
Some scrapers handle JavaScript (they run a browser or browser-like environment); others do plain HTTP only.
- Non-JavaScript — e.g. requests, curl-cffi. Fast and lightweight; use when the page does not rely on client-side rendering.
-
JavaScript-capable — e.g. Playwright, Scrapling with a browser fetcher. Required for SPAs and pages that need execution. These scrapers can accept browser options such as
page_actions(click, scroll, wait, evaluate, etc.) and other per-scraper settings (e.g.wait_for_selector,javascriptfor evaluation result).
Request fields
Every scrape request supports these fields:
| Field | Type | Description | Default |
|---|---|---|---|
url
*required
|
string (URL) | — | |
method
|
string |
GET
|
|
headers
|
object | — | |
query
|
array | object | — | |
data
|
any | Request body; body type is auto-detected (JSON, form, or raw). | — |
cookies
|
array | object | — | |
timeout_ms
|
integer | number | — | |
browser_type
|
string | — | |
headless
|
boolean | — | |
wait_for_selector
|
string | — | |
wait_for_timeout_ms
|
integer | number | — | |
wait_for_load_state
|
string | — | |
page_actions
|
array | — | |
screenshot
|
boolean | — | |
scraper
|
string |
auto
|
|
scraper_options
|
object |
{}
|
|
proxy
|
any | Proxy config object or proxy type string: datacenter, residential, mobile |
datacenter
|
scraperouter
|
any | — |
Response fields
The scrape response includes these fields:
| Field | Type | Description | Default |
|---|---|---|---|
id
|
uuid | Scrape.id | — |
status_code
|
integer | — | |
final_url
|
string | — | |
headers
|
object |
{}
|
|
content
|
string | — | |
cookies
|
array | object | — | |
errors
|
array | — | |
screenshot_url
|
string | — | |
scraper_data
|
object | — | |
scraperouter
|
any | — |
Scraper options scraper_options
Pass options that only apply to a given scraper via the scraper_options object. Unknown keys are ignored by other scrapers.
For the exact options supported by each scraper, check the scraper’s documentation or the Scrapers page.
Meta under scraperouter
Extended metadata for requests and responses may be provided under a scraperouter key. Use it for:
- Request — Optional hints or tracking (e.g. idempotency keys, trace IDs) that your client sends; the API may preserve or echo them.
-
Response — Extra data the API adds (e.g. timing, selected scraper details, or scraper-specific results like
javascript_result) when available. The exact keys depend on the scraper and API version.
Rely only on the unified fields above for stable behavior; treat scraperouter as optional and extensible.