limit from 1 to 10,000. Crawl needs an API key, see Get credentials.
url as they finish, see Read a crawl. A crawl that waits for a free place also has status and queuePosition. This one read /docs/runtime/s3, /docs/runtime/secrets and /docs/runtime/semver, but not /docs/runtime: the start page must match includePaths too. If you already have the URLs, use Batches.
Scope
Patterns are JavaScript regular expressions of at most 200 characters, and match anywhere in the path:
blog matches /company/blog/post. Start with ^ to anchor, as the example does.
Page content
scrapeOptions takes the Scrape fields for each page, such as "scrapeOptions": { "formats": ["markdown", "links"] }. Without it each page returns markdown. linkStyle, maxChars, offset and query are not allowed. A page gets 20 seconds plus waitFor, or scrapeOptions.timeout when you set it, and never more than the time left in the crawl.
Pacing
A
profile in scrapeOptions also makes the crawl read one page at a time. Send "saveChanges": false in it to avoid that.
Errors
A start page that cannot be read ends the crawl with
status: "failed". Check the URL, or start from another page. Every code is in Errors.
Next
- Read a crawl: status, results and webhooks.
- Describe a crawl: get these fields from a sentence.
- Map: list the URLs first.