What is justcrawl.io?

justcrawl.io is a scraping orchestration platform that routes requests across multiple providers (Bright Data, Oxylabs, Nimble Way, Zyte, Decodo) with automatic fallback, cost optimization, and continuous performance tuning.

The problem

You use multiple scraping providers. Each has different strengths per target domain. Managing which provider to use for which URL, handling failures, and keeping costs low is manual, error-prone work.

How justcrawl solves it

1. Benchmark your providers against your actual target URLs
2. Pick a strategy (best success rate, lowest cost, or maximum reliability)
3. Get optimized workflows generated automatically, one per domain
4. Auto-tune continuously as provider performance changes over time

Architecture overview

Your URLs → justcrawl Scheduler → SQS Queue → Worker
                                                  ↓
                                          DAG Executor
                                     ↙         ↓         ↘
                              Provider A   Provider B   Provider C
                                     ↘         ↓         ↙
                                        Results → S3 / Webhook

• Workflows are visual DAGs (directed acyclic graphs) defining how requests flow through providers
• Smart workflows are auto-generated and continuously optimized based on live performance data
• Schedules run your workflows on a recurring basis (hourly, daily, weekly, monthly)
• Integrations let you push URLs via SQS or webhook, and receive results to S3 or webhook

Next steps

Quickstart Sign up to first scrape in 5 minutes

API Quickstart Submit a job via API in under 5 minutes