Extraction & schemas

Scraping gives you a page's raw HTML. Extraction turns that HTML into structured fields — title, price, rating, availability — so you get JSON instead of markup. This page covers when extraction runs, the page types it understands, how the XPath schemas are discovered, and what the quality score on each result actually means.

How extraction runs

Extraction is a workflow step, not a separate API call. Add an extractor node to your workflow and give it a pageType. Every job that flows through that node gets its HTML parsed into the fields for that page type.

The first time anyone scrapes a new (domain, pageType) pair, justcrawl discovers the XPath schema for it automatically — an LLM reads a sample of the page and works out which selectors map to which field. That happens in the background; there is no "discover schema" button or endpoint. Once the schema exists, every later scrape of that domain reuses it instantly, with no LLM step.

Extraction is opt-in per workflow

A workflow with no extractor node returns raw HTML only. Extraction runs only on the jobs you route through an extractor. See Workflows for how to wire one in.

Page types

The pageType you set on the extractor node tells justcrawl what shape of data to pull. It is set by you (it defaults to product); justcrawl does not guess it from the URL.

| pageType | What it extracts | |---|---| | product | Single product: title, price, currency, brand, rating, review_count, availability, description, images, sku | | product_list | A category/grid of products (same fields, repeated) | | serp | Search results: results, result_urls, result_descriptions, total_results, related_searches | | article | Editorial: title, author, publish_date, content, images, tags | | job_posting | Job listing: title, company, location, salary, description, job_type | | bestseller | Ranked list: category_breadcrumb plus index-aligned arrays — product_ranks, product_titles, product_brands, product_prices, product_currencies, product_urls |

Each page type carries its own field template and validation rules, so a price is checked as a number and a rating is bounded to 0–5.

How schemas are discovered

Discovery is LLM-driven and runs on justcrawl's side — the LLM cost is on us, not billed to your credits. A fast model (Claude Haiku) does the first pass; if it can't reliably map at least half the fields on a sample, justcrawl retries with a stronger model (Claude Sonnet). For tricky domains, justcrawl batches a few sample pages together so the discovery sees more than one layout before committing to selectors.

You can read any discovered schema, including how it was found (llm, llm_multi_sample, manual, or reverse_engineered):

# Latest schema for a domain + page type
curl -H "Authorization: Bearer YOUR_API_KEY" \
  "https://dashboard.justcrawl.io/api/v1/extraction/schemas/walmart.com/product"

A 404 here just means nobody has scraped that (domain, pageType) yet — run a job through an extractor and the schema lands shortly after.

The shared schema cache

Extraction schemas are platform-global, not per-org. Once justcrawl has discovered the XPaths for (walmart.com, product), every org that scrapes Walmart product pages reuses the same schema. The practical effect: popular domains extract immediately, with no first-scrape LLM delay for you.

A schema describes page structure only — which selector points at the price, where the title lives. It contains none of your scraped data, none of your URLs, and no signal of which org discovered or uses it. Listing schemas shows you which domains the platform has structure for; it never exposes another customer's results.

# List schemas the platform has discovered (optionally filter by domain)
curl -H "Authorization: Bearer YOUR_API_KEY" \
  "https://dashboard.justcrawl.io/api/v1/extraction/schemas?domain=walmart.com"

Anything you customize for your own org (see escape hatch below) stays private to your org and is merged on top of the shared schema only for your jobs.

Quality scores

Every extraction result carries three scores, each 0–100:

| Score | Meaning | |---|---| | completeness | Share of the page type's fields that came back non-null and passed a sanity check | | validation | Share of those filled fields that passed their type rules (a price is numeric, a rating is 0–5) | | composite | The average of the two — the single number to watch |

The sanity check in completeness is the subtle part: a field that's present but obviously wrong does not count as filled. If a review count lands in the rating field, that field is treated as empty, not as a valid value. This is why a result that looks fully populated can still score below 100 — justcrawl is scoring whether the data is right, not just whether something is there.

You see the scores on every result, and you can dry-run a candidate XPath set against a page you already scraped (this runs synchronously and changes nothing):

curl -X POST https://dashboard.justcrawl.io/api/v1/extraction/test-xpath \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "pageType": "product",
    "attributes": [
      { "name": "price", "type": "number", "xpaths": ["//span[@data-testid=\"price\"]/text()"] }
    ]
  }'

Response:

{
  "values": { "price": 19.99 },
  "qualityScore": { "completeness": 100, "validation": 100, "composite": 100 },
  "jobId": "job-7f3e2a1b-...",
  "url": "https://example.com/product/123"
}

test-xpath uses HTML from a job you already ran (the most recent one for the domain, or a specific jobId). It's capped at 5 MB of HTML per call.

Fixing a low score

Usually you do nothing. A schema that starts returning low-quality results — because the site changed its markup — repairs itself: after a handful of consecutive low scores, justcrawl automatically re-runs discovery and publishes a fresh schema for that domain. The composite score climbing back up is the signal it worked. A low score does not block delivery; results still flow to your outputs while the heal happens.

If you'd rather not wait, or you want fields the standard page type doesn't cover, you can override the schema for your org only. Supply a custom attribute set per domain; same-name attributes replace the platform's XPath, new names extend the result shape:

curl -X PUT https://dashboard.justcrawl.io/api/v1/extraction/attributes/example.com \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "pageType": "product",
    "attributes": [
      { "name": "warranty", "type": "text", "xpaths": ["//div[@id=\"warranty\"]/text()"] }
    ]
  }'

Your overrides merge on top of the shared schema only for your jobs and never touch other orgs. DELETE the same path to revert to the platform schema.

Reading vs. editing needs different permissions

Listing schemas and reading results need jobs:read. Testing XPaths and saving custom attributes need workflows:write. See Roles, permissions & API keys.

Related

• Workflows — add an extractor node to turn extraction on
• Job lifecycle — a result polled during the extraction_done interim returns 202 + Retry-After until indexing finishes
• Roles, permissions & API keys — the jobs:read vs workflows:write split above
• Billing & Plans — discovery LLM cost is platform-absorbed, not billed to your credits