Idempotency

Current behavior

The justcrawl API does not deduplicate requests today. Specifically:

POST /api/v1/jobs with the same URL twice creates two jobs, charges two credits, and triggers two independent scrapes.
POST /api/v1/schedules/{id}/trigger called twice fires two runs.
POST /api/v1/urls/upload with the same CSV twice creates duplicate URL rows (deduplicated by url per workflow, but the upload itself is not idempotent — counts and credits still apply).

There is no Idempotency-Key header today. Sending one is a no-op — the request is processed normally and the header is ignored.

How to retry safely today

Until the header lands, the safe pattern is client-side dedup.

Dedupe by tag

Attach a tag of your own ID to every submitted job. On retry, list jobs filtered by that tag first; submit only if zero matches.

# Generate a stable client-side ID per logical request
CLIENT_ID="order-12345-scrape"

# Check first — is there already a job with this tag?
EXISTING=$(curl -s -H "Authorization: Bearer $JUSTCRAWL_API_KEY" \
  "https://dashboard.justcrawl.io/api/v1/jobs?tag=$CLIENT_ID&limit=1" \
  | jq -r '.jobs[0].id // empty')

if [ -n "$EXISTING" ]; then
  echo "Job already exists: $EXISTING"
else
  curl -X POST https://dashboard.justcrawl.io/api/v1/jobs \
    -H "Authorization: Bearer $JUSTCRAWL_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"url\":\"https://example.com/product/123\",\"tags\":[\"$CLIENT_ID\"]}"
fi

The list-then-submit pattern is not atomic — two parallel retries can both miss the existing job and create duplicates. If you need exactly-once delivery, gate the submit on your side with a local lock or a database row keyed by your client ID.

When to retry vs not

| Response | Retry? | Why | |---|---|---| | 2xx | No. | Success. | | 4xx (except 429 / 408) | No. | Request was wrong; retrying with the same payload won't help. Fix and resubmit. | | 408 Request Timeout | Yes, with dedup. | The request may or may not have been received. | | 429 Too Many Requests | Yes, after a backoff. | See Rate limits for what we send today. | | 5xx | Yes, with dedup. | Transient on our side. | | Network error / connection refused | Yes, with dedup. | The request may or may not have reached us. |

For 408, 5xx, and network errors, the request may have already created a job server-side even though you got an error. Always combine retries with the dedup pattern above, or you'll double-charge yourself on the retry path.

Backoff

Recommended exponential backoff for retried requests:

| Attempt | Wait before retry | |---|---| | 1 | 1s | | 2 | 2s | | 3 | 4s | | 4 | 8s | | 5+ | give up, surface to operator |

Cap retries at 5. Beyond that, the failure is sticky and a human needs to look.

What changes when `Idempotency-Key` ships

The planned design (subject to change):

You send Idempotency-Key: <your-uuid> on any POST.
The server stores the response keyed by (orgId, idempotencyKey) for a fixed replay window (likely 24h).
Subsequent requests with the same key return the original response (same status code, same body, same request_id) without re-executing the side effect.
Keys are scoped per-org — two different orgs can use the same key string without collision.
Sending the same key with a different request body returns a 409 Conflict with code: idempotency_key_reuse.

When this ships, the client-side dedup pattern above becomes unnecessary for new code. Existing code keeps working — the header is additive.

Rate limits — current quota behavior and the planned X-RateLimit-* headers
Webhook delivery — outbound webhooks already use at-least-once delivery; dedupe on jobId
Job lifecycle — how a job moves through the state machine after submit