Rate limits & quotas

Coming soon

Per-org / per-key token-bucket rate limiting with X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, and Retry-After headers is in design but not yet shipped. Today the API enforces credit quotas only — credit-consuming requests are gated on whether you have credits, not on requests-per-second. The behavior documented below is the current reality.

Current behavior — credit quota gate

Every credit-consuming POST (job submit, schedule trigger, URL upload) passes through a quota check before it reaches the handler. The check looks at three things:

1. Is the org's trial still active (if on the trial plan)?
2. Are there remaining credits on the plan?
3. (Belt-and-suspenders) The service layer re-checks and atomically deducts.

If the gate rejects, you get a 402 Payment Required:

{
  "error": "QUOTA_EXCEEDED",
  "message": "Credits exhausted. Contact your administrator to recharge credits.",
  "plan": "starter",
  "remainingCredits": 0,
  "isTrialExpired": false
}

When the trial expires, the message changes to Your trial has expired. Contact your administrator to recharge credits. and isTrialExpired: true.

GET requests always pass the quota gate — you can read your data even when credits are exhausted.

What does NOT exist today

| Feature | Status | |---|---| | X-RateLimit-Limit / -Remaining / -Reset response headers | Not shipped. | | 429 Too Many Requests responses | Not emitted by the gateway. (You may still see 429 from upstream providers in your job result body — that's the provider, not us.) | | Retry-After header | Not shipped. | | Per-key rate limits | Not shipped. Quota is per-org, not per-API-key. | | Per-endpoint quotas | Not shipped. All credit-consuming POSTs share the org credit pool. | | Burst budget separate from sustained rate | Not shipped. |

If your client is built to read X-RateLimit-* or Retry-After headers, those reads will return undefined today. Don't crash on missing.

How to pace yourself today

Without server-side rate-limit signals, the pragmatic approach is:

• Cap your own concurrency. Don't dispatch more than ~50 concurrent job submits per org. The API will accept the burst, but you'll saturate your own credit pool faster than you mean to.
• Watch the 402. If you start getting QUOTA_EXCEEDED, stop submitting and route to a human — retrying without a credit top-up is wasted effort.
• Check credits ahead of large batches. GET /api/v1/plans/current returns your remainingCredits. If you're about to submit 10,000 URLs and have 500 credits, you already know the outcome.

REMAINING=$(curl -s -H "Authorization: Bearer $JUSTCRAWL_API_KEY" \
  https://dashboard.justcrawl.io/api/v1/plans/current \
  | jq -r '.remainingCredits')

echo "You have $REMAINING credits."

What changes when token-bucket rate-limiting ships

The planned design (subject to change):

• Per-org sustained rate (e.g. 100 req/s) and burst budget (e.g. 500 tokens).
• X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset on every response.
• 429 Too Many Requests with Retry-After: <seconds> when the bucket is empty.
• Distinct buckets per-key for power users who want to subdivide their org's allotment.

The credit-quota gate stays — token-bucket controls rate, credit quota controls total spend. They're independent gates.

Related

• Idempotency — how to retry safely today and what Idempotency-Key will look like
• Billing & Plans — how credits map to scrape jobs and plan tiers