Smart Workflows

Smart workflows are generated during onboarding based on your benchmark results. They’re continuously optimized as the system monitors real scraping performance.

How they work

1. During onboarding, you benchmark your target URLs against all connected providers
2. You pick a strategy (success, cost, or reliability)
3. justcrawl generates one workflow per domain where provider ordering differs from the default
4. A default (*) workflow handles all other domains
5. All workflows are published and linked to a strategy

Read-only

Smart workflows cannot be edited in the visual editor. This prevents accidental changes that would break the optimization loop.

To take manual control of a domain, click Clone as Custom. This creates an editable copy with the same DAG. The clone is not smart, it won’t auto-optimize.

Auto vs suggest mode

Each smart workflow has a mode:

Mode	Behavior
Suggest	The engine creates a pending suggestion. You review and apply or dismiss.
Auto	The engine applies changes automatically. You can review the history anytime.

Toggle the mode from the workflow editor page using the Auto-optimize switch.

What the engine monitors

Every 15 minutes, the suggestions engine:

1. Reads live routing scores from the analytics database
2. Compares current provider ordering to optimal ordering
3. If the optimal ordering is different AND the improvement exceeds the threshold:
   • Suggest mode: creates a pending suggestion
   • Auto mode: updates the workflow and publishes immediately

Improvement thresholds

Strategy	Threshold
Success / Reliability	5 percentage points improvement in primary provider success rate
Cost	10% cost reduction

Below these thresholds, the engine skips the cycle to avoid churn.

Next steps

• Strategies for how provider ranking works
• Suggestions for the suggestion lifecycle