Jobs

A job is a single scraping request for one URL, executed through a workflow.

Job lifecycle

Every job moves through a fixed state machine. The short version:

pending → running → extraction_done → completed
                 → failed

| Status | Meaning | |--------|---------| | pending | Created, waiting for a worker | | running | Worker is executing the workflow DAG | | waiting_retry | Backing off before re-trying a provider call | | extraction_done | Extracted, waiting for fan-out ACKs | | completed | All destinations notified; result available | | failed | DAG exhausted or invalid |

For the full state machine — including the extraction_done interim that explains why a job can be "done" but extraction_results may not be queryable for a moment — see Job lifecycle.

Submitting jobs

Via API:

curl -X POST https://dashboard.justcrawl.io/api/v1/jobs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Via schedule: Jobs are created automatically when a schedule runs.

Via external queue: Push {"url": "https://example.com"} to your configured SQS queue or webhook.

Execution trace

Each job records which nodes in the DAG were executed, in order. The trace shows:

• Which provider was tried
• Whether it succeeded or failed
• Status code, latency, error type (if any)

View the trace in the job detail page or via GET /api/v1/jobs/:id.

Getting results

curl https://dashboard.justcrawl.io/api/v1/jobs/JOB_ID/result \
  -H "Authorization: Bearer YOUR_API_KEY"

Returns the scraped HTML as text/html. Only available for completed jobs.

Retry behavior

When a service node in the DAG fails, the worker follows the fail edge to the next provider. This is automatic fallback, not manual retry.

If the entire DAG is exhausted (all providers failed), the job is marked as failed. The SQS visibility timeout handles re-delivery for transient failures.

Quota

Jobs consume credits. Each job costs 1 credit. Check your credit balance at Settings > Billing or via GET /api/v1/plans/status.

When credits are exhausted, job submission returns HTTP 402.