Idempotency

Current behavior

The justcrawl API does not deduplicate requests today. Specifically:

POST /api/v1/jobs with the same URL twice creates two jobs, charges two credits, and triggers two independent scrapes. (A job whose crawl fails at every provider has its credit returned, so a duplicate that fails everywhere costs nothing — but a duplicate that succeeds costs a full credit. Don’t rely on the refund as a dedup mechanism.)
POST /api/v1/schedules/{id}/trigger called twice fires two runs.
POST /api/v1/urls/upload with the same CSV twice creates duplicate URL rows (deduplicated by url per workflow, but the upload itself is not idempotent — counts and credits still apply).

There is no Idempotency-Key header today. Sending one is a no-op — the request is processed normally and the header is ignored.

How to retry safely today

Until the header lands, the safe pattern is client-side dedup.

Dedupe by tag

Attach a tag of your own ID to every submitted job. On retry, list jobs filtered by that tag first; submit only if zero matches.

# Generate a stable client-side ID per logical request
CLIENT_ID="order-12345-scrape"

# Check first — is there already a job with this tag?
EXISTING=$(curl -s -H "Authorization: Bearer $JUSTCRAWL_API_KEY" \
  "https://api.justcrawl.io/api/v1/jobs?tag=$CLIENT_ID&limit=1" \
  | jq -r '.jobs[0].id // empty')

if [ -n "$EXISTING" ]; then
  echo "Job already exists: $EXISTING"
else
  curl -X POST https://api.justcrawl.io/api/v1/jobs \
    -H "Authorization: Bearer $JUSTCRAWL_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"url\":\"https://example.com/product/123\",\"tags\":[\"$CLIENT_ID\"]}"
fi

The list-then-submit pattern is not atomic — two parallel retries can both miss the existing job and create duplicates. If you need exactly-once delivery, gate the submit on your side with a local lock or a database row keyed by your client ID.

When to retry vs not

Response	Retry?	Why
`2xx`	No.	Success.
`4xx` (except 429 / 408)	No.	Request was wrong; retrying with the same payload won’t help. Fix and resubmit.
`408` Request Timeout	Yes, with dedup.	The request may or may not have been received.
`429` Too Many Requests	Yes, after a backoff.	See Rate limits for what we send today.
`5xx`	Yes, with dedup.	Transient on our side.
Network error / connection refused	Yes, with dedup.	The request may or may not have reached us.

For 408, 5xx, and network errors, the request may have already created a job server-side even though you got an error. Always combine retries with the dedup pattern above, or you’ll double-charge yourself on the retry path.

Backoff

Recommended exponential backoff for retried requests:

Attempt	Wait before retry
1	1s
2	2s
3	4s
4	8s
5+	give up, surface to operator

Cap retries at 5. Beyond that, the failure is sticky and a human needs to look.

What changes when `Idempotency-Key` ships

The planned design (subject to change):

You send Idempotency-Key: <your-uuid> on any POST.
The server stores the response keyed by (orgId, idempotencyKey) for a fixed replay window (likely 24h).
Subsequent requests with the same key return the original response (same status code, same body, same request_id) without re-executing the side effect.
Keys are scoped per-org — two different orgs can use the same key string without collision.
Sending the same key with a different request body returns a 409 Conflict with code: idempotency_key_reuse.

When this ships, the client-side dedup pattern above becomes unnecessary for new code. Existing code keeps working — the header is additive.

Rate limits — current quota behavior and the planned X-RateLimit-* headers
Webhook delivery — outbound webhooks already use at-least-once delivery; dedupe on jobId
Job lifecycle — how a job moves through the state machine after submit