Skip to main content
A batch reads a list of URLs in the background, with the same scrape options for every page.
Send 1 to 10,000 URLs of at most 2,048 characters each. Each page comes back as Markdown, and an address sent twice is read once. Batches need an API key, see Get credentials.
The call answers at once. invalidURLs lists the addresses that are not public website URLs: they are skipped and the rest runs. A batch that waits for a free place also has status and queuePosition. Read the pages from url as they finish, see Batch results. If you want the URLs found for you, use Crawl.

Fields

The API reference has every field.

Add URLs to a running batch

Send the batch id in appendToId and the new addresses in urls. The batch keeps its own options and webhook, skips URLs it already has, and counts the new ones in total. Added URLs count toward the 10,000 cap.

Retry a submit safely

Send an Idempotency-Key header with any text. The same key and body within 24 hours return the first answer with idempotent-replayed: true. The same key with a different body starts a new batch.

Errors

A batch ends with status: "failed" when your balance runs out or no page could be read. completed means every page was tried, so check errors. Every code is in Errors.

Next

  • Batch results: read, retry, cancel and list batches.
  • Scrape: the options every page takes.
  • Map: list a site’s URLs to send.