Integrations

Integrations connect justcrawl to your existing infrastructure for real-time URL ingestion and result delivery.

Input sources

SQS queue

Push URLs to your own SQS queue. justcrawl polls it every few seconds and creates jobs automatically.

Message format:

{"url": "https://example.com/product/123"}

Optionally include a workflowId to override domain-based routing:

{"url": "https://example.com/product/123", "workflowId": "workflow-uuid"}

If no workflowId is provided, justcrawl resolves the workflow by domain (same as scheduled jobs).

Setup: Settings > Integrations > Add Input Source, or during onboarding. Provide your AWS Access Key ID, Secret Access Key, Region, and Queue URL.

Webhook ingestion

justcrawl generates a unique webhook URL. POST an array of URLs to it:

curl -X POST https://dashboard.justcrawl.io/api/v1/webhooks/YOUR_TOKEN/ingest \
  -H "Content-Type: application/json" \
  -d '["https://example.com/a", "https://example.com/b"]'

Or with explicit workflow assignment:

[{"url": "https://example.com/a", "workflowId": "workflow-uuid"}]

Daily URL cap applies per input config (default: 10,000/day).

Output destinations

S3 bucket

Scraping results are uploaded to your S3 bucket as HTML files:

s3://your-bucket/prefix/JOB_ID.html

Setup: Provide AWS credentials, bucket name, and optional prefix.

Webhook delivery

Results are POSTed to your endpoint as JSON, signed with HMAC-SHA256 using your shared secret. Three attempts with exponential backoff per delivery; 4xx aborts immediately, 5xx and network errors retry.

For the full deep dive — payload format, signature verification code samples in Node and Python, response-code handling, SSRF protection rules, and operational notes on ordering and at-least-once semantics — see Webhook delivery.

Managing integrations

All integrations are managed in Settings > Integrations. Each config can be enabled/disabled independently. Status indicators show the last successful poll/delivery time and any errors.