Skip to main content
Batches are read like crawls: polling, next paging, the status fields, errors, maxTextLength and webhooks work as in Read a crawl, at /v1/batch/scrape. A batch and its pages are kept for 24 hours, then reading it returns 404 notFound. These calls are not charged.
A page that answers 404 or 410 counts as completed, with metadata.error and no charge. failed counts pages that could not be read. Each page in data is a scrape answer.

Calls

The older creditsUsed field is a legacy field. Use costUsd.

Retry failed pages

Read /errors and take the url of each row with retryable: true. Send them in urls to a new POST /v1/batch/scrape with the same options, or to the running batch with appendToId. Wait a few seconds before retrying sourceUnavailable and sourceRefused. Do not retry a row with retryable: false, such as notFound, siteNotRead or promptInjectionDetected. Codes are in Errors and retry timing in Troubleshooting. Batches share job limits with crawls: see Rate limits.