Providers

justcrawl.io routes scraping requests across multiple providers. You bring your own API keys.

Supported providers

Provider	Auth	Auto-route by URL	Async webhook	Vendor docs
Bright Data	API Key + Zone	No (single zone per account)	No	docs.brightdata.com
Oxylabs	Username + Password	Yes (33 verified source templates)	No (TODO 2)	developers.oxylabs.io
Nimble Way	Bearer token	No (driver-based stealth)	No	docs.nimbleway.com
Zyte	API Key	No (auto-extract is TODO 3)	No	docs.zyte.com
Decodo	API Key (formerly Smartproxy)	Yes (17 verified target templates)	No (TODO 2)	help.decodo.com

Connecting a provider

1. Go to Settings > Provider Accounts
2. Click Add Provider
3. Enter your credentials
4. Click Test Connection to verify
5. Save

Credentials are encrypted at rest with AES-256-GCM and never returned in API responses.

Provider pricing

In Settings > Vendor Pricing, enter what you pay per 1,000 requests for each provider. This enables:

• Cost strategy: ranks providers by cheapest-first
• Cost estimates: shows expected cost per workflow on the dashboard
• Premium domains: optionally set higher rates for specific domains (e.g., social media sites)

Multiple providers required

Smart optimization (benchmarking, strategy selection, auto-tuning) requires at least 2 connected providers. With a single provider, justcrawl acts as a simple proxy with scheduling and URL management.

---

Per-node settings

Every service node in a workflow accepts a shared set of settings (Country, JS rendering, Wait selector, Parse, Screenshot, Markdown, Custom Headers, Custom Cookies, Sessions, Device, Advanced JSON) plus vendor-specific extras. See Service node settings for the full reference.

The capability matrix below shows which providers support which features. Cells marked ✓ are supported and exposed in the workflow editor.

Feature	Bright Data	Oxylabs	Nimble Way	Zyte	Decodo
JS rendering	✓ (always)	✓	✓	✓	✓
Geo-targeting	✓	✓	✓	✓	✓
Wait for selector	✓	✓	✓	✓	✓
Parse (vendor-side structured)	—	✓	—	TODO 3	✓
Screenshot (PNG)	✓	✓	✓	✓	✓
Markdown output	✓	—	✓	—	✓
Custom headers	✓	✓	✓	✓	✓
Custom cookies	—	✓	✓	✓	✓
Sessions (sticky IP)	—	✓	—	✓	✓
Device emulation	—	✓	✓	✓	✓
Auto-route by URL	—	✓	—	—	✓
Advanced (JSON) passthrough	✓	✓	✓	✓	✓

---

Bright Data

Bright Data Web Unlocker is a synchronous proxy service that automatically renders JavaScript and handles anti-bot detection. The adapter targets the Web Unlocker product specifically. Async datasets (Web Scraper API) and SERP API are out of scope (TODO 1).

Required configuration

• API Key — sent as Authorization: Bearer <key>.
• Zone — the Web Unlocker zone name from your Bright Data dashboard (e.g., web_unlocker1). Must be Web Unlocker type, not Web Scraper, SERP, or Browser. Auto-detected from your account at connection time; pick from the dropdown.

Capabilities

Feature	Notes
JS rendering	Always on. Web Unlocker can’t be disabled.
Geo-targeting	Lowercase ISO-2 (us, gb, de). Passed as-is.
Wait for selector	Encoded in x-unblock-expect header. Requires “Manual expect elements” enabled on zone.
Screenshot	Set screenshot: true → data_format: "screenshot". Body is base64 PNG, contentType: image/png.
Markdown	Set markdown: true → data_format: "markdown". Body is markdown text. Screenshot wins if both set.
Custom headers	Merged into request body headers field (preserves x-unblock-expect). Requires “Custom Headers & Cookies” enabled on zone.
Sessions / cookies / device emulation	Not exposed by Web Unlocker.

Response format

• format: "json" (default): {status_code, body, headers} envelope. body contains the rendered HTML or base64 PNG / markdown if a data_format is set.
• format: "raw": response is the body directly, no envelope.

Gotchas

Zone-type mismatch. If your zone is not Web Unlocker, the response shape is different and body is missing. The adapter detects this and returns errorType: 'validation' with the message Bright Data response missing 'body' field — likely zone-type mismatch (zone must be Web Unlocker, not Web Scraper / SERP / Browser).

Premium domains. Walmart, Target, Instacart, and similar require a separate premium zone in your Bright Data account. Standard zones fail.

Feature gates. customHeaders and waitForSelector only work if the corresponding feature is enabled on the zone in the Bright Data UI. Without the feature, the field is silently ignored.

Vendor docs

• Web Unlocker overview
• Configuration
• Quickstart
• Features
• API reference
• SERP API (separate product, not wired)

Zone routing

When you save Bright Data credentials, ScrapeRoute calls GET /zone/get_active_zones and stores the zones (with their type) in org_provider_resources. A periodic job (sync_provider_resources, every 15 min) keeps that table in sync with the dashboard. At scrape time, the resolver picks a zone in this order:

1. Per-node zone override (workflow editor → service node config). Non-empty value wins over everything below.
2. Host detector — known SERP hosts route through your SERP-typed zone: google.*, bing.com, duckduckgo.com, yandex.*, and yahoo.com/search. If your account doesn’t have a SERP zone, the resolver falls through.
3. credentials.zone — your account’s default Web Unlocker zone.

The workflow editor’s zone dropdown includes a Use account default option (empty value) that lets a node opt out of an explicit pick and rely on the detector + credentials default.

What’s not supported

• Web Scraper API async (datasets/v3/trigger with snapshot polling) — TODO 1.

---

Oxylabs

Oxylabs Realtime Web Scraper API. The adapter exposes 33 verified source templates (amazon_product, google_search, walmart, ebay, etsy, kroger, target, bestbuy, …) — every entry verified against developers.oxylabs.io — plus a universal fallback for any URL.

Auto-routing by URL (headline feature)

Leave the Source field empty and the adapter auto-detects the right Oxylabs source and parameter shape from your URL:

URL	Routes to	With params
amazon.com/dp/B08Y72CH1F	amazon_product	{query: "B08Y72CH1F", domain: "com"}
amazon.co.uk/dp/B0XXXX	amazon_product	{query: "B0XXXX", domain: "co.uk"}
amazon.com/s?k=laptop	amazon_search	{query: "laptop", domain: "com"}
walmart.com/ip/widget/123456	walmart_product	{product_id: "123456", domain: "com"}
walmart.com/search?q=tv	walmart_search	{query: "tv"}
google.com/search?q=test	google_search	{query: "test", domain: "com"}
ebay.com/itm/...	ebay	{url}
any other host	universal	{url}

International TLDs are preserved (amazon.com.br → domain: "com.br"). Detector source: libraries/vendors/providers/src/oxylabs-source-detector.ts.

To override, set Source explicitly in the node config — that wins over auto-detection. See the legacy-row guard below if you do.

Required configuration

• Username + Password — HTTP Basic auth.

Capabilities

Feature	Notes
JS rendering	renderJs: true → render: "html".
Geo-targeting	ISO-2 lowercase converted to full country name (us → "United States").
Wait for selector	browser_instructions[0] with CSS wait. Requires JS rendering. Default timeout 10s.
Parse	parse: true → Oxylabs returns structured JSON, contentType: application/json.
Screenshot	screenshot: true → render: "png". Base64 PNG. Wins over renderJs.
Custom headers	Forwarded via context: [{key: "headers", value: {...}}].
Custom cookies	Flat Record<string,string> → context: [{key: "cookies", value: {...}}].
Sessions	sessionId → context: [{key: "session_id", value: ...}].
Device	desktop / mobile / tablet → user_agent_type (tablet maps to mobile).
HTTP POST	httpMethod: "POST" + httpBody → context: [{key: "http_method", value: "post"}, {key: "content", value: "<base64>"}].

Legacy-row guard

Oxylabs’s per-source identifier-field contract is heterogeneous, not uniform. Sending the wrong field name returns HTTP 400 with a vendor message like [product_id]: This field is missing. [query]: This field was not expected.

Identifier field	Sources	Param shape
url	universal, amazon, walmart, ebay, google, …	{url}
query (+ optional domain)	amazon_product, amazon_search, walmart_search, google_search	{query, domain?}
product_id (+ optional domain)	walmart_product	{product_id, domain?}

If you set Source explicitly without supplying the matching field in sourceParams, the adapter returns:

errorType: 'validation'
error: "Oxylabs source 'walmart_product' requires 'product_id' in sourceParams. See https://developers.oxylabs.io/scraping-solutions/web-scraper-api/targets/walmart/product. Or clear the source field to use auto-detection from the URL."

Workaround: clear Source to use auto-detection, or populate sourceParams with the field that matches your chosen source ({query, domain} for search-style, {product_id, domain} for product-style) in Advanced (JSON).

Gotchas

Render timeout cap. The adapter’s maxTimeoutMs is 30s; Oxylabs’s upstream cap is 180s. Use Advanced (JSON) to extend (e.g., {"timeout": 120000}) up to the upstream cap.

Parsed = JSON, not HTML. When parse: true, the body is JSON.stringify-ed parsed data with contentType: application/json. Downstream HTML validators skip this content type automatically.

Per-source identifier reference

Oxylabs’s per-source contract is heterogeneous: URL-based sources take url, search-style sources take query (+ optional domain), product-style sources take product_id (+ optional domain). Sending the wrong field name fails with HTTP 400 and a vendor message like [product_id]: This field is missing. [query]: This field was not expected. The detector picks the right field for you when you leave Source on Auto-detect from URL; the table below is the contract for explicit-source workflows.

Identifier-field legend:

• url — pass the full URL in sourceParams.url (or leave the URL field unchanged for auto).
• query — pass a search term in sourceParams.query.
• product_id — pass the vendor’s product id (e.g., Walmart item id, Amazon ASIN) in sourceParams.product_id.

Generated from SOURCE_REQUIRED_PARAM in libraries/vendors/providers/src/oxylabs-source-detector.ts. Run pnpm exec tsx scripts/generate-vendor-doc-tables.ts to refresh.

Source	Group	Identifier field	Vendor docs
universal	General	url	link
amazon	Amazon	url	link
amazon_product	Amazon	query	link
amazon_search	Amazon	query	link
amazon_pricing	Amazon	query	link
amazon_bestsellers	Amazon	query	link
amazon_sellers	Amazon	query	link
google	Google	url	link
google_search	Google	query	link
google_ads	Google	query	link
google_travel_hotels	Google	query	link
google_lens	Google	query	link
google_shopping_search	Google	query	link
google_shopping_product	Google	query	link
google_trends_explore	Google	query	link
bing	Bing	url	link
bing_search	Bing	query	link
walmart	Walmart	url	link
walmart_product	Walmart	product_id	link
walmart_search	Walmart	query	link
ebay	eBay	url	link
ebay_search	eBay	query	link
etsy	E-commerce	url	link
kroger	E-commerce	url	link
target	E-commerce	url	link
bestbuy	E-commerce	url	link
lowes	E-commerce	url	link
costco	E-commerce	url	link
aliexpress	E-commerce	url	link
alibaba	E-commerce	url	link
rakuten	E-commerce	url	link
flipkart	E-commerce	url	link
mercadolibre	E-commerce	url	link

Vendor docs

• Web Scraper API overview
• Targets index (full source list)
• Custom Parser (use Advanced (JSON) with parsing_instructions)
• JavaScript rendering
• Integration methods (Realtime / Proxy / Push-Pull)
• Forming requests

What’s not supported

• Push-Pull async webhook — TODO 2. Realtime endpoint only today.
• parsing_instructions UI — available via Advanced (JSON) only.

---

Nimble Way

Nimble Way Web API. URL-uniform — there is no auto-routing by URL. Site-aware behavior comes from explicit driver selection (Nimble’s stealth profiles).

Required configuration

• API Token — sent as Authorization: Bearer <token>.

Driver-based stealth

Nimble doesn’t have site-specific templates. Instead, drivers tune scraping behavior:

Driver	Use case
vx6	Fast, no JavaScript — plain HTML only
vx8	Headless + JS rendering (default when renderJs: true or waitForSelector set)
vx8-pro	Enhanced headless variant
vx10	Full-page rendering variant
vx10-pro	Stealth headful — auto-selected from device: 'mobile' or device: 'tablet'
Auto (default)	Inferred per request: device hint → JS flag → vx6 fallback

When Driver is empty, the adapter infers:

• device: 'mobile' | 'tablet' → vx10-pro (with render enabled)
• renderJs: true or waitForSelector → vx8
• otherwise → backend default

Capabilities

Feature	Notes
JS rendering	render: true + driver. Auto-enabled with renderJs, waitForSelector, or device: 'mobile'/'tablet'.
Geo-targeting	Uppercase ISO-2 (US, GB, FR). Sub-country via Advanced (JSON).
Wait for selector	browser_actions array with timeout (default 10s). Forces vx8.
Screenshot	formats: ['html', 'screenshot']. Base64 PNG, contentType: image/png.
Markdown	formats: ['html', 'markdown']. contentType: text/markdown.
Custom headers	Pass-through object → body.headers.
Custom cookies	Flat map → array of {key, value, domain}. Each cookie’s domain defaults to the request URL hostname if not specified.
Device	desktop / mobile / tablet → driver mapping above.

Gotchas

device: 'desktop' alone is a no-op. Selecting Desktop without also enabling renderJs or waitForSelector does not trigger rendering — Nimble requires render: true to be paired with a driver. Combine with Render JS or a wait selector. Fixed in v0.3.4.

Auto driver fix in v0.3.4. Empty driver string previously short-circuited inference. Now device: 'mobile' still resolves to vx10-pro and renderJs: true still falls back to vx8 when Driver = Auto.

Cookies must include domain. The adapter auto-fills it from the request URL hostname; only override via Advanced (JSON) if you need a different domain (e.g., .example.com).

Vendor docs

• API overview
• Web Tools Extract quickstart
• API reference
• Web API quickstart
• Authentication

What’s not supported

• Endpoint migration — adapter currently uses sdk.nimbleway.com/v1/extract. The canonical endpoint per docs is api.webit.live/api/v1/realtime/web. Migration is on the roadmap; both currently route the same.
• First-class sessions — no session ID; identity is per-request.
• parsing schema UI — available via Advanced (JSON) with a CSS-selector schema.

---

Zyte

Zyte API at /v1/extract. Two execution modes — browser and HTTP — with a strict mutual-exclusion guard at the adapter boundary.

Required configuration

• API Key — HTTP Basic with key as username, empty password.

Two execution modes

Browser mode — full headless Chrome. Triggered when you set renderJs, screenshot, or waitForSelector.

• Sends browserHtml: true to Zyte; supports actions array (waitForSelector with timeout).
• For headers: only Referer is honored on browser mode (other custom headers are silently dropped server-side).

HTTP mode — pure fetch. Default when no browser flag is set.

• Sends httpResponseBody: true to Zyte. Adapter base64-decodes the body to UTF-8.
• Supports POST body and full custom-headers passthrough.
• Required for httpMethod: "POST" and device emulation.

Mutual exclusion. Combining browser flags (renderJs, screenshot, waitForSelector) with HTTP-mode options (httpMethod: "POST" or httpBody) returns errorType: 'validation' before reaching Zyte’s API. Pick one mode.

Capabilities

Feature	Notes
JS rendering	Browser mode (browserHtml: true).
Geo-targeting	Uppercase ISO-2 → geolocation. 20 countries free; 50+ extended tier costs extra.
Wait for selector	actions[0] with timeout — adapter converts ms → seconds.
Screenshot	screenshot: true → base64 PNG. Adds extra cost. Browser mode only.
Custom headers	HTTP mode: customHttpRequestHeaders (full passthrough). Browser mode: requestHeaders (only Referer honored).
Custom cookies	requestCookies array. Each cookie’s domain defaults to the request URL hostname per eng-review A2.
Sessions	session.id (RFC 4122 v4 UUID). Free-form strings are deterministically hashed to a stable v4 UUID.
Device	HTTP mode only. tablet maps to mobile. Silently omitted in browser mode.
HTTP POST	HTTP mode only. UTF-8 body via httpRequestText.
ipType	Vendor-specific node field: datacenter (default, free) or residential (KYC + per-request surcharge).

Gotchas

Selector timeout unit. justcrawl uses milliseconds; Zyte uses seconds. The adapter converts (Math.round(ms / 1000)). Default 10000 ms → timeout: 10 seconds.

Screenshot is independent of renderJs. Setting screenshot: true triggers browser mode on its own without a redundant browserHtml: true. Set both if you want HTML and a screenshot in the same request.

Residential IPs require KYC. Zyte requires identity verification before allowing residential routing. Per-request surcharge applies. Defaults to datacenter.

Sessions UUID coercion. Zyte requires session.id to be a v4 UUID. The adapter SHA-1-hashes any free-form string (e.g., "user-123-cart") to a deterministic v4 UUID — same input always produces the same Zyte session.

Vendor docs

• API reference (top-level params, mutual exclusion)
• Browser mode (actions, viewport, device)
• HTTP mode
• Auto-extraction (product / article / jobPosting — TODO 3)
• Features (geolocation, ipType, sessions, cookies)
• Proxy mode

What’s not supported

• Auto-extraction routing (product: true, article: true, jobPosting: true) — TODO 3. The flags can be set via Advanced (JSON), but the response routing for structured JSON is not wired into the validation pipeline yet. Use only if you control the downstream consumer.

---

Decodo

Decodo Web Scraping API real-time mode (formerly Smartproxy). 17 verified target templates (every entry checked against help.decodo.com’s llms.txt index) with auto-routing similar to Oxylabs. URLs that don’t match a verified target route through universal.

Auto-routing by URL

Leave Target empty for automatic routing. Same shape as Oxylabs:

URL	Routes to	With params
amazon.com/dp/B08Y72CH1F	amazon_product	{query: "B08Y72CH1F", domain: "com"}
amazon.com/s?k=laptop	amazon_search	{query: "laptop", domain: "com"}
amazon other paths	amazon	{url}
google.com/search?q=test	google_search	{query: "test", domain: "com"}
google other paths	universal	{url}
walmart.com/ip/widget/12345	walmart_product	{product_id: "12345"}
walmart.com/search?q=tv	walmart_search	{query: "tv"}
walmart other	walmart	{url}
ebay.*	universal	{url} (Decodo ships no per-target eBay doc page; URL passthrough)
any other host	universal	{url}

Detector source: libraries/vendors/providers/src/decodo-target-detector.ts. Explicit Target wins over auto-detection.

Required configuration

• API Key (token) — sent as HTTP Basic username with empty password.

Capabilities

Feature	Notes
JS rendering	renderJs: true → headless: "html".
Geo-targeting	ISO-2 → Title Case country name (e.g., us → "United States").
Wait for selector	browser_actions[0] with default 10s timeout.
Parse	parse: true → vendor-side parsed JSON (e.g., product details), contentType: application/json.
Screenshot	screenshot: true → headless: "png". Base64 PNG. Wins over markdown / renderJs.
Markdown	markdown: true → markdown text + auto-enables headless rendering.
Custom headers	body.headers object passthrough.
Custom cookies	body.cookies object passthrough.
Sessions	session_id for sticky sessions.
Device	device_type: desktop / mobile / tablet.
HTTP POST	http_method: "POST" + payload (base64).

Decodo-specific node fields

Field	Notes
Target	Explicit override — 17 verified options grouped by site. Empty = auto-detect.
Proxy Pool	standard (default) or premium.
Page From / Page Count	Only meaningful for paginated targets like google_search, amazon_search.

Legacy-row guard

Decodo mirrors Oxylabs’s heterogeneous identifier-field contract. Sending the wrong field name returns HTTP 400.

Identifier field	Targets	Param shape
url	universal, amazon, walmart, google	{url}
query (+ optional domain)	amazon_product, amazon_search, walmart_search, google_search	{query, domain?}
product_id	walmart_product	{product_id}

If you set Target explicitly without supplying the matching field in targetParams:

errorType: 'validation'
error: "Decodo target 'walmart_product' requires 'product_id' in targetParams. See https://help.decodo.com/docs/web-scraping-api-walmart-product. Or clear the target field to use auto-detection from the URL."

Gotchas

150s server timeout. Decodo’s real-time hard cap is 150s. The adapter uses maxTimeoutMs: 30000 as a conservative client guard.

Screenshot wins over markdown. If you set both, you get the PNG.

Markdown forces headless. The adapter auto-enables headless: "html" when markdown: true.

Per-target identifier reference

Decodo’s target contract mirrors Oxylabs’s: URL-based targets take url, search-style take query (+ optional domain), product-style take product_id (+ optional domain). The detector picks the right field for you when you leave Target on Auto-detect from URL; the table below is the contract for explicit-target workflows. Decodo’s documented surface is narrower than Oxylabs’s — only targets with a verified vendor doc page are listed; URLs that don’t match any of these go through universal.

Generated from TARGET_REQUIRED_PARAM in libraries/vendors/providers/src/decodo-target-detector.ts. Run pnpm exec tsx scripts/generate-vendor-doc-tables.ts to refresh.

Target	Group	Identifier field	Vendor docs
universal	General	url	link
amazon	Amazon	url	link
amazon_product	Amazon	query	link
amazon_search	Amazon	query	link
amazon_pricing	Amazon	query	link
amazon_bestsellers	Amazon	query	link
amazon_sellers	Amazon	query	link
google	Google	url	link
google_search	Google	query	link
google_ads	Google	query	link
google_travel_hotels	Google	query	link
google_shopping_search	Google	query	link
google_shopping_product	Google	query	link
bing_search	Bing	query	link
walmart	Walmart	url	link
walmart_product	Walmart	product_id	link
walmart_search	Walmart	query	link

Vendor docs

• Web Scraping API introduction
• Quick start
• Real-time requests
• Asynchronous requests (v3/task) — not wired (TODO 2)
• API parameters
• Browser actions

What’s not supported

• v3/task async API — TODO 2. Real-time only today.