Analytics

The analytics dashboard shows scraping performance across all providers, domains, and time ranges.

Overview metrics

Top-level cards showing:

• Total jobs in the selected time range
• Success rate across all providers
• Average latency in milliseconds
• Total cost (estimated from vendor pricing)

Timeseries

Line charts showing metrics over time. Supports ranges from 1 hour to 30 days. Intervals adjust automatically (5m for 1h, 1h for 24h, 1d for 30d).

Provider comparison

Side-by-side comparison of all connected providers:

• Success rate per provider
• Average latency per provider
• Job count per provider
• Cost per 1,000 requests

Useful for identifying which provider is underperforming and whether smart workflow optimization is making the right decisions.

Domain breakdown

Per-domain metrics showing which domains are hardest to scrape and which providers perform best for each domain. This is the same data the suggestions engine uses.

Routing scores

The routing scores table shows per-(domain, provider) performance used for smart workflow optimization:

• Success rate
• Sample count
• Average latency
• Last computed time

Scores are recomputed every 15 minutes from live scraping data in TimescaleDB.

Provider health

Real-time health indicators per provider:

• Current success rate (last 1 hour)
• Trend (improving, stable, degrading)
• Last successful scrape timestamp

API access

All analytics data is available via the API:

# Overview
curl https://dashboard.justcrawl.io/api/v1/analytics/overview?hours=24 \
  -H "Authorization: Bearer YOUR_API_KEY"

# Timeseries
curl https://dashboard.justcrawl.io/api/v1/analytics/timeseries?hours=24&metric=success_rate \
  -H "Authorization: Bearer YOUR_API_KEY"

# Provider comparison
curl https://dashboard.justcrawl.io/api/v1/analytics/providers?hours=24 \
  -H "Authorization: Bearer YOUR_API_KEY"