Workflows

A workflow is a visual DAG (directed acyclic graph) that defines how a scraping request moves through your providers.

Node types

Node	Purpose
Entry	Starting point. Every workflow has exactly one.
Service	Calls a scraping provider (Bright Data, Oxylabs, etc.). Configured with a provider ID.
Split	Distributes traffic across multiple paths by percentage (e.g., 50/50 A/B test).
Result	Terminal node for successful scrapes.
Failed	Terminal node for failed scrapes (all providers exhausted).

Edge types

Edge	Meaning
Default	Standard flow (Entry to first node, Split to branches).
Success	Taken when a service node succeeds.
Fail	Taken when a service node fails (triggers fallback).

Fallback chains

The most common pattern: Entry → Provider A → (fail) → Provider B → (fail) → Provider C → Failed.

Each service node’s success edge goes to Result. The fail edge goes to the next provider. The last provider’s fail edge goes to Failed.

Smart workflows generate this pattern automatically based on benchmark data.

Publishing and versioning

Workflows have versions. Each save creates a new version. Only published workflows are used by the scheduler and job submission.

• Draft: editable, not used for scraping
• Published: active, used for new jobs
• Unpublishing reverts to draft

Domain routing

Workflows can have a route that determines which URLs they handle:

• domain:amazon.com matches URLs on amazon.com
• * is the default fallback for unmatched domains

When a job is submitted, justcrawl matches the URL’s domain against published workflow routes. The most specific match wins.

Smart vs custom workflows

Smart workflows are generated during onboarding and continuously optimized. They’re read-only in the editor. To customize, clone a smart workflow as a custom draft.

Custom workflows are built manually in the visual editor. Full control over node placement, provider selection, and routing logic.