Jobs

A job is a single scraping request for one URL, executed through a workflow.

Job lifecycle

Every job moves through a fixed state machine. The short version:

pending → running → extraction_done → completed
                 → failed

Status	Meaning
pending	Created, waiting for a worker
running	Worker is executing the workflow DAG
waiting_retry	Backing off before re-trying a provider call
extraction_done	Extracted, waiting for fan-out ACKs
completed	All destinations notified; result available
failed	DAG exhausted or invalid

For the full state machine — including the extraction_done interim that explains why a job can be “done” but extraction_results may not be queryable for a moment — see Job lifecycle.

Submitting jobs

Via API:

curl -X POST https://api.justcrawl.io/api/v1/jobs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Via schedule: Jobs are created automatically when a schedule runs.

Via external queue: Push {"url": "https://example.com"} to your configured SQS queue or webhook.

Execution trace

Each job records which nodes in the DAG were executed, in order. The trace shows:

• Which provider was tried
• Whether it succeeded or failed
• Status code, latency, error type (if any)

View the trace in the job detail page or via GET /api/v1/jobs/:id.

Getting results

curl https://api.justcrawl.io/api/v1/jobs/JOB_ID/result \
  -H "Authorization: Bearer YOUR_API_KEY"

Returns the scraped HTML as text/html. Only available for completed jobs.

Retry behavior

When a service node in the DAG fails, the worker follows the fail edge to the next provider. This is automatic fallback, not manual retry.

If the entire DAG is exhausted (all providers failed), the job is marked as failed. The SQS visibility timeout handles re-delivery for transient failures.

Quota

Jobs consume credits. Each job costs 1 credit, taken when the job is submitted. Check your credit balance at Settings > Billing or via GET /api/v1/plans/status.

When credits are exhausted, job submission returns HTTP 402.

A crawl that fails at every provider is free: you only pay when a provider delivers a page. When a job ends failed and no provider delivered a page, its credit is returned automatically and the job says so. See Billing & Plans.