Skip to content
justcrawl.io

Webhook delivery

When you configure an output with outputType: webhook, every completed scrape POSTs its result to the URL you registered. This page covers the wire format, how to verify the signature, what happens when your endpoint is slow or returns an error, and the SSRF defenses you'll trip if you point at a private address.

For the broader output-config story (S3 vs webhook vs warehouse), see Integrations.

Setup

Section titled “Setup”
Terminal window
curl -X POST https://dashboard.justcrawl.io/api/v1/integrations/outputs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Webhook to data team",
    "outputType": "webhook",
    "destination": "https://data.example.com/justcrawl-events",
    "credentials": { "hmacSecret": "your-32-byte-shared-secret" }
  }'

Two things to know:

  1. destination must be a public URL. SSRF defense rejects localhost, RFC 1918 private ranges, IPv6 link-local / ULA, the 169.254.169.254 AWS metadata IP, and any hostname ending in .internal or .local. The block runs at submit time, then runs again on every delivery (DNS can change). See SSRF protection below.
  2. hmacSecret is encrypted at rest with AES-256-GCM and is never returned in any API response. Store your copy somewhere safe — if you lose it, mint a new one via PUT /outputs/{id} and re-register the secret on your receiver.

What you receive

Section titled “What you receive”

Every delivery is a POST with these headers:

| Header | Value | |---|---| | Content-Type | application/json | | X-JustCrawl-Signature | sha256=<hex> — HMAC-SHA256 of the raw request body using your hmacSecret | | X-JustCrawl-Job-Id | The job ID, also present in the body. Useful for log correlation before you parse the body. |

The body is JSON:

{
  "jobId": "job-7f3e2a1b-...",
  "url": "https://example.com/product/123",
  "statusCode": 200,
  "providerId": "brightdata",
  "latencyMs": 1247,
  "body": "<html>...</html>",
  "deliveredAt": "2026-06-21T14:32:08.142Z"
}

| Field | Notes | |---|---| | jobId | Same value as the X-JustCrawl-Job-Id header. | | url | The URL that was scraped. | | statusCode | The provider's HTTP status. null if the scrape never reached a provider (e.g., validation rejected the response). | | providerId | Which provider returned the body — brightdata, oxylabs, nimble, zyte, or decodo. | | latencyMs | Wall-clock time from job submit to provider return. | | body | The raw HTML response body, inlined as a string. Large pages produce large payloads — receivers should accept request bodies of at least 10 MB. | | deliveredAt | When we built this payload, in ISO 8601 UTC. Not the time the scrape ran. |

Verify the signature

Section titled “Verify the signature”

The signature is HMAC-SHA256 of the raw request body bytes (not the parsed object), hex-encoded, prefixed with sha256=. Verify on every request — a request without a valid signature should be rejected with 401.

Node.js (Express):

import crypto from 'node:crypto';
import express from 'express';


const app = express();
const SECRET = process.env.JUSTCRAWL_HMAC_SECRET;


// IMPORTANT: capture the raw body before any JSON parser touches it.
app.use(express.raw({ type: 'application/json', limit: '20mb' }));


app.post('/justcrawl-events', (req, res) => {
  const received = req.header('X-JustCrawl-Signature') || '';
  const expected = 'sha256=' + crypto
    .createHmac('sha256', SECRET)
    .update(req.body)
    .digest('hex');


  // Constant-time compare to prevent timing attacks.
  if (
    received.length !== expected.length ||
    !crypto.timingSafeEqual(Buffer.from(received), Buffer.from(expected))
  ) {
    return res.status(401).send('Invalid signature');
  }


  const payload = JSON.parse(req.body.toString('utf8'));
  // ... do your work ...
  res.status(200).send('ok');
});

Python (Flask):

import hmac, hashlib, os
from flask import Flask, request, abort


app = Flask(__name__)
SECRET = os.environ['JUSTCRAWL_HMAC_SECRET'].encode()


@app.post('/justcrawl-events')
def receive():
    received = request.headers.get('X-JustCrawl-Signature', '')
    expected = 'sha256=' + hmac.new(SECRET, request.data, hashlib.sha256).hexdigest()


    if not hmac.compare_digest(received, expected):
        abort(401)


    payload = request.get_json()
    # ... do your work ...
    return 'ok', 200

Retry behavior

Section titled “Retry behavior”

Each delivery makes up to 3 attempts with exponential backoff. Per-attempt timeout is 30 seconds. Redirects (3xx) are not followed — your destination URL must be the final one.

| Attempt | Delay before attempt | |---|---| | 1 | 0s | | 2 | 2s after attempt 1 fails | | 3 | 4s after attempt 2 fails |

If all three attempts fail with retryable errors (see below), the underlying queue retries the whole sequence again after its visibility timeout. Repeated retryable failures eventually land the message in a dead-letter queue, which we monitor.

Response codes

Section titled “Response codes”

How we classify your endpoint's response:

| Your response | We treat it as | What happens | |---|---|---| | 2xx | Success | We move on. Delivery acknowledged. | | 4xx | Permanent failure | We stop retrying immediately. The message goes to DLQ. This includes 401, 404, and 410 — there is no special reconnect handshake; if your endpoint moved or rotated its secret, update the output config via PUT /outputs/{id}. | | 5xx | Transient failure | Retry the next attempt. | | Network error / timeout / connection refused | Transient failure | Retry the next attempt. |

SSRF protection

Section titled “SSRF protection”

Every webhook URL is validated on submit and on every delivery. We reject:

  • localhost, 0.0.0.0
  • IPv4 private ranges: 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12
  • IPv4 link-local: 169.254.0.0/16 (including AWS instance metadata at 169.254.169.254)
  • IPv6 loopback ::1, link-local fe80::/10, unique local fc00::/7
  • IPv4-mapped IPv6 of any of the above
  • Hostnames ending in .internal or .local
  • Hostnames that fail DNS resolution

DNS resolution checks every A and AAAA record, not just the first one. This closes the DNS-rebind window where a hostname with both a public and private record passes the initial check, then resolves to the private record on the actual delivery.

If your URL is blocked, the delivery is classified as a permanent failure (same as 4xx) and lands in DLQ.

Pausing and disabling

Section titled “Pausing and disabling”

To stop deliveries to an endpoint without losing the config:

Terminal window
curl -X PATCH https://dashboard.justcrawl.io/api/v1/integrations/outputs/OUTPUT_ID/toggle \
  -H "Authorization: Bearer YOUR_API_KEY"

While paused, in-flight messages drain (we'll still attempt them), but no new deliveries are queued. Toggle again to resume.

To delete the config permanently, use DELETE /outputs/{id}. Deletion is hard — there's no soft-delete or grace period for webhook configs (the system-managed internal output is the only one we refuse to delete).

Operational notes

Section titled “Operational notes”
  • Throughput. The webhook fan-out polls every 5 seconds and batches up to 100 deliveries per cycle per writer. A single hot webhook URL is rate-limited only by your endpoint's response time, not by us.
  • Ordering. We do not guarantee delivery order. Two jobs that complete in the same second may arrive at your endpoint in either order. Use jobId for idempotency; don't rely on deliveredAt being monotonic.
  • At-least-once delivery. The queue is at-least-once — a network blip during the ACK back to us can cause us to re-deliver a payload your endpoint already accepted. Deduplicate on jobId.
Section titled “Related”
  • Integrations — the broader integrations story (SQS input, S3 output, webhook input vs output)
  • Job lifecycle — webhooks fire as part of the fan-out that promotes a job to completed