Skip to content
justcrawl.io

Rate limits & quotas

Current behavior — credit quota gate

Section titled “Current behavior — credit quota gate”

Every credit-consuming POST (job submit, schedule trigger, URL upload) passes through a quota check before it reaches the handler. The check looks at three things:

  1. Is the org’s trial still active (if on the trial plan)?
  2. Are there remaining credits on the plan?
  3. (Belt-and-suspenders) The service layer re-checks and atomically deducts.

If the gate rejects, you get a 402 Payment Required:

{
  "error": "QUOTA_EXCEEDED",
  "message": "Credits exhausted. Contact your administrator to recharge credits.",
  "plan": "starter",
  "remainingCredits": 0,
  "isTrialExpired": false
}

When the trial expires, the message changes to Your trial has expired. Contact your administrator to recharge credits. and isTrialExpired: true.

GET requests always pass the quota gate — you can read your data even when credits are exhausted.

Credits returned for crawls that failed at every provider go straight back into the pool, and the next submit’s check honours them. See Billing & Plans.

What does NOT exist today

Section titled “What does NOT exist today”
Feature Status
X-RateLimit-Limit / -Remaining / -Reset response headers Not shipped.
429 Too Many Requests responses Partially shipped — see below. No general rate limiter emits 429, but two endpoint-specific caps do. (You may also see 429 from upstream providers in your job result body — that’s the provider, not us.)
Retry-After header Partially shipped — see below. Not sent on the two 429s; it is sent on one 202.
Per-key rate limits Not shipped. Quota is per-org, not per-API-key.
Per-endpoint quotas Not shipped. All credit-consuming POSTs share the org credit pool.
Burst budget separate from sustained rate Not shipped.

X-RateLimit-* reads return undefined today — don’t crash on missing.

The two 429s and one Retry-After that DO exist

Section titled “The two 429s and one Retry-After that DO exist”

There is no general rate limiter, but three specific responses will surprise a client built on the table above. All three are current as of 2026-08-28.

Endpoint Status When Retry-After?
POST /api/v1/bi/queries 429 Too many concurrent BI queries for the org (too_many_queries) No
POST /api/v1/webhooks/{token}/ingest 429 Daily URL cap exceeded; the message carries the limit, today’s count, and the requested count No
GET /api/v1/extraction/results/{jobId} 202 Extraction finished but is still being indexed — body carries indexingPending: true and retryAfterSeconds Yes, Retry-After: 5

Two consequences for a client:

  • Treat 429 as retryable on idempotent reads. Neither 429 sends Retry-After, so back off on your own schedule. Do not retry the POSTs themselves without checking whether the first one took effect — there is no Idempotency-Key support yet.
  • 202 is not a result. On GET /api/v1/extraction/results/{jobId}, branch on indexingPending before reading result fields, and honor Retry-After (or the equivalent retryAfterSeconds body field) rather than polling on a fixed interval.

How to pace yourself today

Section titled “How to pace yourself today”

Without server-side rate-limit signals, the pragmatic approach is:

  • Cap your own concurrency. Don’t dispatch more than ~50 concurrent job submits per org. The API will accept the burst, but you’ll saturate your own credit pool faster than you mean to.
  • Watch the 402. If you start getting QUOTA_EXCEEDED, stop submitting and route to a human — retrying without a credit top-up is wasted effort.
  • Check credits ahead of large batches. GET /api/v1/plans/current returns your remainingCredits. If you’re about to submit 10,000 URLs and have 500 credits, you already know the outcome.
Terminal window
REMAINING=$(curl -s -H "Authorization: Bearer $JUSTCRAWL_API_KEY" \
  https://api.justcrawl.io/api/v1/plans/current \
  | jq -r '.remainingCredits')


echo "You have $REMAINING credits."

What changes when token-bucket rate-limiting ships

Section titled “What changes when token-bucket rate-limiting ships”

The planned design (subject to change):

  • Per-org sustained rate (e.g. 100 req/s) and burst budget (e.g. 500 tokens).
  • X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset on every response.
  • 429 Too Many Requests with Retry-After: <seconds> when the bucket is empty — extending Retry-After to the two 429s documented above.
  • Distinct buckets per-key for power users who want to subdivide their org’s allotment.

The credit-quota gate stays — token-bucket controls rate, credit quota controls total spend. They’re independent gates.

Section titled “Related”
  • Idempotency — how to retry safely today and what Idempotency-Key will look like
  • Billing & Plans — how credits map to scrape jobs and plan tiers