Skip to content
justcrawl.io

Switch storage between platform and customer S3 bucket

PUT
/api/v1/integrations/storage
Code sample: cURL 
curl -X PUT 'https://dashboard.justcrawl.io/api/v1/integrations/storage' \
  -H 'Authorization: Bearer $JUSTCRAWL_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"type":"custom","bucket":"my-scrape-results","accessKeyId":"AKIA...","secretAccessKey":"...","region":"us-east-1","documentExpiryDays":90}'

Switches where this org’s job result blobs are written. Two paths:

  • type: "platform" — reset to JustCrawl-managed storage. Bucket and credentials fields are ignored. Side-effect free; the existing encrypted bucket credentials (if any) are wiped from the DB.
  • type: "custom" — switch to a customer-owned S3 bucket. Requires bucket + accessKeyId + secretAccessKey + region. Credentials are validated synchronously by uploading then deleting a test object at .justcrawl-test-<timestamp> against the supplied bucket — if that fails, the call returns 400 with the AWS error and does not persist anything. On success the bucket’s GetBucketVersioning is read to set storageVersioningEnabled (drives the _v1/_v2 suffix fallback); credentials are AES-256-GCM encrypted at rest.

Optional documentExpiryDays (1–3650, or null for indefinite) drives the justcrawl-retention S3 Lifecycle rule installed best-effort (existing customer lifecycle rules with a different ID are preserved). If the lifecycle PUT fails (IAM permission gap), the response includes a lifecycleWarning field — the storage swap still succeeds. If the bucket has versioning disabled, the response includes a versioningWarning field.

Requires the org:manage permission.

Authorizations

Section titled “Authorizations ”

Request Body required

Section titled “Request Body required ”
Media type application/json
object
type
required
string
Allowed values: platform custom
bucket

Required when type=custom. S3 bucket name.

string
accessKeyId

Required when type=custom. Validated by a PutObject + DeleteObject roundtrip.

string
secretAccessKey

Required when type=custom. AES-256-GCM encrypted at rest.

string
region

Required when type=custom. AWS region (e.g. us-east-1).

string
documentExpiryDays

1–3650 inclusive, or null for indefinite. Drives the bucket’s S3 Lifecycle rule (best-effort).

integer
nullable

Responses

Section titled “ Responses ”

200

Section titled “200 ”

New storage config. Includes optional versioningWarning and lifecycleWarning fields when applicable.

Media type application/json
object
type
string
Allowed values: platform custom
bucket
string
nullable
storageVersioningEnabled
boolean
versioningWarning
string
lifecycleWarning
string
Example 
{
  "type": "platform"
}

400

Section titled “400 ”

Validation failed — request body or query is missing/invalid

Media type application/json
object
error
string
Example 
{
  "error": "Missing or invalid \"url\" field"
}

401

Section titled “401 ”

Missing or invalid authentication token

Media type application/json
object
error
string
Example 
{
  "error": "Missing or invalid authentication token"
}

403

Section titled “403 ”

Insufficient permissions for this operation

Media type application/json
object
error
string
Example 
{
  "error": "No organization. Complete onboarding first."
}

500

Section titled “500 ”

Unexpected server error. Logs and PostHog $exception capture

Media type application/json
object
error
string
Example 
{
  "error": "Something went wrong"
}