Quickstart

Get from zero to a working scraping pipeline in one onboarding flow.

Step 1: Sign up

Create an account at dashboard.justcrawl.io/register. You get a 7-day free trial with 1,000 jobs. No credit card required.

Step 2: Create your organization

Enter your organization name. A URL-safe slug is generated automatically.

Step 3: Connect providers

Select which scraping providers you use (Bright Data, Oxylabs, Nimble Way, Zyte, Decodo). You need at least 2 providers to unlock smart optimization.

For each provider, enter your API credentials. justcrawl tests the connection immediately.

Step 4: Enter pricing

Enter what you pay per 1,000 requests for each provider. This enables the cost optimization strategy. Skip if you don’t know your rates.

Step 5: Benchmark

Paste your target URLs (up to 20). justcrawl tests each URL against every connected provider 3 times, measuring success rate, latency, and cost.

Watch results come in real time. If a provider’s credentials are invalid, it’s flagged immediately.

Step 6: Pick a strategy

Choose what matters most:

Strategy	What it does
Best Success Rate	Routes to the highest-performing provider per domain. Fallback on failure.
Lowest Cost	Routes to the cheapest provider meeting a quality threshold.
Maximum Reliability	Chains all providers as fallbacks for zero downtime.

Step 7: Review workflows

justcrawl generates one workflow per domain (e.g., amazon.com gets a different provider order than walmart.com) plus a default workflow for unknown domains. All are published automatically.

Step 8: Set up schedule

Choose how often to run (hourly, daily, weekly, monthly). Pick a time and timezone. Your benchmark URLs are saved to your URL library automatically.

Step 9: Optional integrations

Optionally configure an SQS queue or webhook for real-time URL ingestion, and S3 or webhook for result delivery.

Done

You’re on the dashboard with published workflows, saved URLs, and an active schedule. Scraping starts at the next scheduled run.