> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl

> Start from one URL, follow its links in the background, and collect each page as Markdown or any scrape format. Set the scope by path, depth and topic.

Crawl reads a site's pages in the background, starting from one URL and following its links.

```bash cURL theme={null}
curl -X POST https://api.stophy.dev/v1/crawl \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://bun.com/docs/runtime",
    "includePaths": ["^/docs/runtime/s"],
    "limit": 3
  }'
```

```typescript TypeScript theme={null}
import { Stophy } from "stophy";

const stophy = new Stophy();

const { id } = await stophy.crawl({
  url: "https://bun.com/docs/runtime",
  includePaths: ["^/docs/runtime/s"],
  limit: 3,
});
console.log(id);
```

```python Python theme={null}
from stophy import Stophy

stophy = Stophy()

result = stophy.crawl(
    url="https://bun.com/docs/runtime",
    include_paths=["^/docs/runtime/s"],
    limit=3,
)
print(result["id"])
```

A crawl stays under the start page's path on its host and reads at most 10,000 pages. Set `limit` from 1 to 10,000. Crawl needs an API key, see [Get credentials](https://docs.stophy.dev/get-credentials.md).

```json theme={null}
{
  "success": true,
  "id": "job_MUETS-T9eC87YeCL49pTOA",
  "url": "https://api.stophy.dev/v1/crawl/job_MUETS-T9eC87YeCL49pTOA"
}
```

The call answers at once. Read the pages from `url` as they finish, see [Read a crawl](https://docs.stophy.dev/crawl/async.md). A crawl that waits for a free place also has `status` and `queuePosition`. This one read `/docs/runtime/s3`, `/docs/runtime/secrets` and `/docs/runtime/semver`, but not `/docs/runtime`: the start page must match `includePaths` too. If you already have the URLs, use [Batches](https://docs.stophy.dev/batches/overview).

## Scope

| Field | Default | What it does |
| - | - | - |
| `includePaths`, `excludePaths` | `[]` | Regular expressions on the path, at most 100 each. `excludePaths` wins. |
| `regexOnFullURL` | `false` | Match the patterns against the whole URL. |
| `maxDiscoveryDepth` | none | How many links from the start page to follow. |
| `crawlEntireDomain` | `false` | Read the whole site, not only the start page's path. |
| `allowSubdomains` | `false` | Also read subdomains such as `docs.bun.com`. |
| `allowExternalLinks` | `false` | Also follow links to other sites. They count toward `limit` and depth. |
| `sitemap` | `include` | `include` also uses sitemaps, `only` reads what they name, `skip` ignores them. |
| `ignoreQueryParameters` | `false` | Treat URLs that differ only in `?query` as one page. |
| `deduplicateSimilarURLs` | `true` | Read one page for URLs that differ only by `www`, a trailing slash, a `#fragment`, tracking parameters such as `utm_source`, or `?query` order. |
| `prompt` | none | A topic, at most 1,000 characters. Pages about it are read first, and the crawl stops once new pages stop mentioning it. |

Patterns are JavaScript regular expressions of at most 200 characters, and match anywhere in the path: `blog` matches `/company/blog/post`. Start with `^` to anchor, as the example does.

## Page content

`scrapeOptions` takes the [Scrape](https://docs.stophy.dev/scrape/overview) fields for each page, such as `"scrapeOptions": { "formats": ["markdown", "links"] }`. Without it each page returns `markdown`. `linkStyle`, `maxChars`, `offset` and `query` are not allowed. A page gets 20 seconds plus `waitFor`, or `scrapeOptions.timeout` when you set it, and never more than the time left in the crawl.

## Pacing

| Field | What it does |
| - | - |
| `maxConcurrency` | Pages read at once, 1 to 100. A crawl reads at most 4 at once. |
| `delay` | Seconds between pages, at most 60. The crawl then reads one page at a time. |
| `webhook` | Be told as the crawl runs. See [Webhooks](https://docs.stophy.dev/webhooks.md). |
| `zeroDataRetention` | Keep nothing from the crawl. See [Zero data retention](https://docs.stophy.dev/production/zero-data-retention.md). |

A `profile` in `scrapeOptions` also makes the crawl read one page at a time. Send `"saveChanges": false` in it to avoid that.

## Errors

| Code | Cause | Fix |
| - | - | - |
| `400 invalidRequest` | A pattern that is not valid, or that repeats a group with `*`, `+` or `{}`, such as `(ab)+` | Fix the pattern that `error` names |
| `429 rateLimited` | More jobs running or waiting than your tier allows | Wait `retryAfterSeconds`. See [Rate limits](https://docs.stophy.dev/production/rate-limits.md). |

A start page that cannot be read ends the crawl with `status: "failed"`. Check the URL, or start from another page. Every code is in [Errors](https://docs.stophy.dev/api-reference/errors.md).

## Next

* [Read a crawl](https://docs.stophy.dev/crawl/async.md): status, results and webhooks.
* [Describe a crawl](https://docs.stophy.dev/crawl/describe.md): get these fields from a sentence.
* [Map](https://docs.stophy.dev/map/overview): list the URLs first.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.