Skip to main content
Crawl reads a site’s pages in the background, starting from one URL and following its links.
A crawl stays under the start page’s path on its host and reads at most 10,000 pages. Set limit from 1 to 10,000. Crawl needs an API key, see Get credentials.
The call answers at once. Read the pages from url as they finish, see Read a crawl. A crawl that waits for a free place also has status and queuePosition. This one read /docs/runtime/s3, /docs/runtime/secrets and /docs/runtime/semver, but not /docs/runtime: the start page must match includePaths too. If you already have the URLs, use Batches.

Scope

Patterns are JavaScript regular expressions of at most 200 characters, and match anywhere in the path: blog matches /company/blog/post. Start with ^ to anchor, as the example does.

Page content

scrapeOptions takes the Scrape fields for each page, such as "scrapeOptions": { "formats": ["markdown", "links"] }. Without it each page returns markdown. linkStyle, maxChars, offset and query are not allowed. A page gets 20 seconds plus waitFor, or scrapeOptions.timeout when you set it, and never more than the time left in the crawl.

Pacing

A profile in scrapeOptions also makes the crawl read one page at a time. Send "saveChanges": false in it to avoid that.

Errors

A start page that cannot be read ends the crawl with status: "failed". Check the URL, or start from another page. Every code is in Errors.

Next