Job lifecycle

Every job you submit to justcrawl.io moves through a fixed state machine. Most jobs complete in seconds and you never see the intermediate states. But when something is slow, retrying, or stuck, knowing which state the job is in tells you where to look.

This page covers the full lifecycle. For the basics of submitting jobs and fetching results, see Jobs.

State machine

            ┌────────────────────────────────────────────┐
            ▼                                            │
        pending ──► running ──► waiting_retry ───────────┘
                        │
                        ├──► extraction_done ──► completed
                        │
                        └──► failed

A job always starts in pending and ends in either completed or failed. The path between them depends on what the workflow does and whether any provider hops fail along the way.

The states

pending

The job row exists, but no worker has picked it up yet. The URL is sitting in our internal queue, partitioned per-org so one busy customer can't starve another's traffic.

How long it lasts: typically under a second. If you see a job stuck in pending for more than a minute, the most likely cause is your org's per-org inflight cap being saturated by other jobs running concurrently. The cap defaults to 5 and is configured per org by us; reach out if your traffic shape needs it raised.

running

A worker has claimed the job and is walking the workflow DAG. This covers:

• Calling the first provider (Bright Data, Oxylabs, Nimble Way, Zyte, or Decodo)
• Uploading the response body to storage
• Running extraction if the workflow has an extractor node
• Following fail edges to fallback providers if the primary returned an error

Most of a job's wall-clock time lives here. The detail page shows a live execution trace as nodes complete.

waiting_retry

A provider returned a retryable error (typically a 5xx or a timeout) and the workflow's service node has retries > 0 configured. The worker has scheduled a deferred retry with backoff. When the timer fires, the job moves back to running and the provider call is re-attempted.

Retry configuration

Retries are configured per service node, not per job. Each service node accepts retries (max attempts), retryDelayMs (delay before the next attempt), and retryBackoff (instant or deferred).

extraction_done

This is the state that surprises people, so it gets its own section. See The extraction_done interim below.

completed

The job is done. All extraction results have been written, every configured destination has been notified (the internal Postgres store, customer webhooks, BigQuery / Snowflake / Databricks warehouses, and — on the BI plan — ClickHouse), and GET /api/v1/jobs/{id}/result will return the scraped body.

This is the only state in which extraction_results and extraction_deliveries rows are guaranteed to be readable for this job.

failed

The job will not produce a result. Common causes:

• Every fallback provider in the DAG returned an error
• The HTML validation gate flagged the response as a CAPTCHA, soft-404, or empty body
• The workflow DAG itself was invalid (no terminal warehouse node — every extraction has to land somewhere)
• Quota was exhausted mid-flight (rare — quota is checked at submit time)

The job's errorType field tells you which class of failure occurred. The execution trace shows exactly which node returned the fail edge.

The extraction_done interim

When a workflow finishes extracting data, the result fans out to one or more destinations in parallel:

• Postgres — the extraction_results row that powers /extractions queries
• Webhooks — outbound POSTs to every enabled webhook output config
• ClickHouse — for orgs on the BI warehouse

Each destination has its own bulk-writer that acknowledges asynchronously. The job sits in extraction_done while we wait for every destination to ACK. Once all ACKs land, the job promotes to completed.

What this means for you:

• A job in extraction_done has successfully extracted data — the workflow did its job.
• The extraction_results row may not be readable yet. If you poll right at the moment a job hits extraction_done, the row appears a moment later (typically under a second).
• Webhook deliveries are likely in flight or just landed.

When you'd notice it:

• Polling GET /api/v1/jobs/{id} in a tight loop and seeing extraction_done before completed.
• A webhook receiver firing within the same second that an extraction_results query returns "not found" — both are downstream of the same fan-out.

What to do about it:

• For one-shot reads, wait for completed before fetching extraction_results. The fan-out is fast; total polling time is dominated by the scrape, not the ACK wait.
• For event-driven flows, listen to the webhook delivery instead of polling. The webhook fires after the bulk-writer ACKs, which is the same gate that promotes the job to completed.

Polling the state

curl https://dashboard.justcrawl.io/api/v1/jobs/JOB_ID \
  -H "Authorization: Bearer YOUR_API_KEY"

The response carries the current status, the full executionTrace, and timestamps for each transition. A reasonable poll cadence is every 2-5 seconds. If you need lower-latency notifications, use webhooks or a warehouse destination — both are pushed at the same moment the job hits completed.

State reference

| Status | What's happening | Terminal? | |---|---|---| | pending | Queued, waiting for a worker | No | | running | Worker is walking the DAG | No | | waiting_retry | Backing off before re-trying a provider call | No | | extraction_done | Extracted, waiting for fan-out ACKs | No | | completed | All destinations notified; results readable | Yes | | failed | DAG exhausted or invalid | Yes |

Related

• Jobs — submitting jobs and fetching results
• Workflows — the DAG that determines which states a job visits
• Service node settings — per-node provider configuration (country, JS rendering, headers, sessions, output format)
• job_terminal — the error returned when you try to mutate a job in completed or failed