Crawl control reference

Start a crawl

POST/crawls

Scope:

crawl

Rate class:

start

Operation id:

start_crawl

Starts a crawl of one of your sites (My Sites) and returns at once with 202 Accepted, the crawl object and a Location header; follow it with GET /crawls/{id}. It goes through the same checks as the dashboard (public addresses only, blocked domains, your plan's rates, the owned-site rate for a verified site).

Credits. Credits are checked before the crawl exists: the most it can cost (max_pages times the page rate) must fit what the workspace has left this month, less what your other running API crawls may still spend (402 insufficient_credits otherwise; credits.available is already net of those holds). You are charged when the crawl ends, for the pages it checked.

Limits. A key's daily_credit_cap bounds the worst-case cost of the crawls it starts per UTC day (429 daily_credit_cap_reached); a key made without one may crawl at most 500 pages. Crawls of one site through the API must be 10 minutes apart (429 crawl_too_soon; a crawl cancelled before it started does not count). A key may start 10 crawls a day and the workspace 20; 2 may run at once. A site that already has a pending or running crawl answers 409 crawl_in_progress with that crawl's id.

Idempotency. Send Idempotency-Key (1 to 255 characters) to make a retry safe: the same key with the same body returns the crawl the first request started (200 and Idempotent-Replayed: true); the same key with another body is 422 idempotency_key_reused; a request still being made answers 409 idempotency_in_progress. Keys are remembered for 24 hours. Use something unique per intended crawl, for example a commit hash.

Needs the crawl scope. Until starting crawls through the API is switched on, this route answers 404.

Parameters

Request body

application/json, required.

{
  "max_pages": 500,
  "mode": "fast",
  "render_js": false,
  "site_id": 12
}

{
  "id": "3f2b8c1e-5d47-4a0b-9a6e-1c7d2e8f4b90",
  "site_id": 12,
  "url": "https://www.example.com",
  "mode": "fast",
  "status": "pending",
  "started_via": "api",
  "created_at": "2026-10-31T09:12:44Z",
  "max_pages": 500,
  "pages_checked": 0,
  "links_checked": 0,
  "broken_count": 0,
  "working_count": 0,
  "processing": {
    "inventory": "pending",
    "fixes": "pending"
  },
  "report_url": "https://dashboard.perfbee.com/#/brokenLinks?scan=3f2b8c1e-5d47-4a0b-9a6e-1c7d2e8f4b90",
  "render_js": false,
  "credits": {
    "per_page": 0.5,
    "max_cost": 250,
    "remaining_before": 8759.5,
    "held_by_running_crawls": 0
  }
}

{
  "id": "3f2b8c1e-5d47-4a0b-9a6e-1c7d2e8f4b90",
  "site_id": 12,
  "url": "https://www.example.com",
  "mode": "fast",
  "status": "pending",
  "started_via": "api",
  "created_at": "2026-10-31T09:12:44Z",
  "max_pages": 500,
  "pages_checked": 0,
  "links_checked": 0,
  "broken_count": 0,
  "working_count": 0,
  "processing": {
    "inventory": "pending",
    "fixes": "pending"
  },
  "report_url": "https://dashboard.perfbee.com/#/brokenLinks?scan=3f2b8c1e-5d47-4a0b-9a6e-1c7d2e8f4b90",
  "render_js": false,
  "credits": {
    "per_page": 0.5,
    "max_cost": 250,
    "remaining_before": 8759.5,
    "held_by_running_crawls": 0
  }
}

Problems this call documents

• 402 Not enough credits: insufficient_credits
• 404 Not found: not_found
• 409 Crawl in progress: crawl_in_progress
• 409 Request in progress: idempotency_in_progress
• 422 Idempotency key reused: idempotency_key_reused
• 422 Validation failed: validation_error
• 429 Daily crawl limit reached: crawl_limit_reached
• 429 Crawl too soon: crawl_too_soon
• 429 Daily credit cap reached: daily_credit_cap_reached
• 429 Too many running crawls: too_many_running_crawls
• 503 Service busy: capacity_busy

Cancel a crawl

POST/crawls/{crawl_id}/cancel

Scope:

crawl

Rate class:

light

Operation id:

cancel_crawl

Asks for one of your workspace's crawls to stop. Only a crawl that is pending or running can be cancelled (409 crawl_not_active otherwise). A crawl that is queued, or waiting between two parts, ends at once; a running crawl is told to stop and the crawler does so within a few seconds, so the answer is 202 Accepted and the crawl's status becomes cancelled shortly after: watch GET /crawls/{id}.

Pages checked so far stay billed. The crawl is charged for the pages it checked before it stopped, as after a cancel in the dashboard; a crawl cancelled before it started costs nothing. Cancelling does not give credits back.

Needs the crawl scope (a viewer's key cannot). Until starting crawls through the API is switched on, this route answers 404.

Parameters

{
  "id": "3f2b8c1e-5d47-4a0b-9a6e-1c7d2e8f4b90",
  "status": "running",
  "cancelled": false,
  "message": "The crawler was told to stop. It saves the pages it checked, which stay billed. Poll the crawl until its status is cancelled."
}

Problems this call documents

• 404 Not found: not_found
• 409 Crawl is not active: crawl_not_active
• 422 Validation failed: validation_error
• 503 Service busy: capacity_busy