Skip to content
justcrawl.io

API Quickstart

For developers who want to use the API directly. Get from API key to scraped HTML in 5 minutes.

Step 1: Get your API key

Section titled “Step 1: Get your API key”

Go to Settings > API Keys in the dashboard and create a new key. Copy the token, it’s only shown once.

Step 2: Submit a job

Section titled “Step 2: Submit a job”
Terminal window
curl -X POST https://dashboard.justcrawl.io/api/v1/jobs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Response:

{
  "id": "job-uuid",
  "status": "queued",
  "url": "https://example.com",
  "createdAt": "2026-04-11T00:00:00Z"
}

Step 3: Poll for completion

Section titled “Step 3: Poll for completion”
Terminal window
curl https://dashboard.justcrawl.io/api/v1/jobs/JOB_ID \
  -H "Authorization: Bearer YOUR_API_KEY"

Response when completed:

{
  "id": "job-uuid",
  "status": "completed",
  "url": "https://example.com",
  "providerId": "brightdata",
  "statusCode": 200,
  "latencyMs": 1250,
  "createdAt": "2026-04-11T00:00:00Z",
  "completedAt": "2026-04-11T00:00:02Z"
}

Possible statuses: queued, running, completed, failed.

Step 4: Get the result

Section titled “Step 4: Get the result”
Terminal window
curl https://dashboard.justcrawl.io/api/v1/jobs/JOB_ID/result \
  -H "Authorization: Bearer YOUR_API_KEY"

Returns the scraped HTML body as text/html.

Batch polling

Section titled “Batch polling”

Poll multiple jobs at once:

Terminal window
curl -X POST https://dashboard.justcrawl.io/api/v1/jobs/batch-status \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"jobIds": ["job-1", "job-2", "job-3"]}'

What happens under the hood

Section titled “What happens under the hood”
  1. justcrawl resolves which workflow to use for your URL (based on domain routing)
  2. The job enters the SQS queue with per-org fairness sharding
  3. The worker executes the workflow DAG: tries the primary provider, falls back on failure
  4. Results are stored and optionally delivered to your S3 bucket or webhook

Next steps

Section titled “Next steps”