> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Read a crawl

> Poll a crawl, page through its results, read its errors, cancel it, list your crawls and get webhooks. Batches use the same calls, so this page covers both.

A crawl runs in the background. These calls read, cancel and list it, and none of them is charged. They need your API key. A crawl and its pages are kept for 24 hours, and an unknown, expired or other account's `id` returns `404 notFound`.

A [batch](https://docs.stophy.dev/batches/overview) works the same way at `/v1/batch/scrape/{id}`, with `/results` and `/errors` under it. This page is the home for the shared behavior.

## Read every page

Poll `GET /v1/crawl/{id}` until `status` is `completed`, `failed` or `cancelled`, then follow `next` until it is missing.

```bash cURL theme={null}
CRAWL_ID="<the id your crawl returned>"
URL="https://api.stophy.dev/v1/crawl/$CRAWL_ID"
while :; do
  BODY=$(curl -s "$URL" -H "Authorization: Bearer $STOPHY_API_KEY")
  case $(echo "$BODY" | jq -r .status) in completed|failed|cancelled) break ;; esac
  sleep 2
done
echo "$BODY" | jq .data
NEXT=$(echo "$BODY" | jq -r '.next // empty')
while [ -n "$NEXT" ]; do
  BODY=$(curl -s "$NEXT" -H "Authorization: Bearer $STOPHY_API_KEY")
  echo "$BODY" | jq .data
  NEXT=$(echo "$BODY" | jq -r '.next // empty')
done
```

```typescript TypeScript theme={null}
import { Stophy } from "stophy";

const stophy = new Stophy();

for await (const page of stophy.crawlResults("<the id your crawl returned>")) {
  console.log(page.metadata?.sourceURL);
}
```

```python Python theme={null}
from stophy import Stophy

stophy = Stophy()

for page in stophy.crawl_results("<the id your crawl returned>"):
    print(page["metadata"]["sourceURL"])
```

A status answer holds up to 100 finished pages or 2 MB, whichever comes first, which is separate from the crawl's `limit`. `next` appears only when more finished pages wait, so it never leads to an empty `data`. It is a full URL: call it with the same key. A `cursor` works only for its own job and `limit`.

The SDK iterators read through a `checkpoint`: `checkpoint=ck_start` starts, and each answer returns the next checkpoint and only pages you have not seen. They wait while the job runs and end when it is over and every page was delivered. After 15 minutes without finishing they stop with `resultsTimeout`. Pass `timeoutMs` (`timeout` in Python) to change that, and `onCheckpoint` (`on_checkpoint`) to resume after a restart. For a batch use `batchScrapeResults` (`batch_scrape_results`).

## Status fields

```json theme={null}
{
  "success": true,
  "data": [],
  "status": "scraping",
  "total": 1,
  "completed": 0,
  "failed": 0,
  "costUsd": 0,
  "expiresAt": "2026-11-18T18:03:49.796Z",
  "errors": []
}
```

| Field | What it holds |
| - | - |
| `status` | `scraping`, `completed`, `failed` or `cancelled`. `failed` means the job stopped, such as an unreadable start page or an empty balance, with the reason in `error`. Failed pages do not make a job fail. |
| `queuePosition` | Where the job waits, before it starts. |
| `total` | Pages found so far. It grows while the crawl runs, so `data` can be empty while `total` is 1. |
| `completed`, `failed` | Pages that finished and pages that could not be read. |
| `data` | Finished pages, in the shape of the formats you chose. |
| `errors` | Failed pages in this answer: `url`, `code`, `message`, `retryable`, `at`. |
| `costUsd` | What the pages read so far cost. |
| `createdAt`, `completedAt`, `duration`, `expiresAt` | Times, and the seconds the job ran. Pages are removed at `expiresAt`. |
| `next` | The next answer, when more pages wait. |

## Results with depth

`GET /v1/crawl/{id}/results` returns finished pages with more detail: the `url` the crawl ended on, the `requestedUrl` it followed, `depth` and a `costUsd` per page. It returns 20 at a time. Send `limit` up to 100 and `cursor` from the answer, with the same `limit`, until `cursor` is missing. A part can hold fewer than `limit`.

## Errors

`GET /v1/crawl/{id}/errors` lists the pages that could not be read, up to 1,000, with `isTruncated` when there were more. `retryable` says whether trying again can help. Error codes and fixes are in [Errors](https://docs.stophy.dev/api-reference/errors.md). `robotsBlocked` lists up to 1,000 links that robots rules disallow. They are skipped, not failures.

A 404 or 410 page is not an error. It is a finished page with its `metadata.statusCode` and a `metadata.error`.

## Cancel and list

* `DELETE /v1/crawl/{id}` cancels a running crawl and keeps the pages already read, which are charged. A finished job returns `409 jobFinished`.
* `GET /v1/crawl` lists your crawls, newest first, 20 at a time. Send `status`, `limit` up to 100 and `cursor`. A part can hold fewer than `limit`.
* `GET /v1/crawl/active` lists crawls that are waiting or running: `id`, `url`, `status` (`queued` or `scraping`), `createdAt` and `options`.

## Webhooks

Send `webhook` when you start a job instead of polling. It is a URL, or an object.

```json theme={null}
{ "webhook": { "url": "https://example.com/hook", "events": ["page", "completed"], "metadata": { "project": "docs" } } }
```

| Field | Effect |
| - | - |
| `url` | Public address that answers `2xx`. Required. |
| `headers` | Sent with every call. For a batch, up to 10, values up to 1,000 characters. |
| `metadata` | Comes back in every call. For a batch, up to 4,000 characters. |
| `events` | Any of `started`, `page`, `completed`, `failed`, `cancelled`, and for a crawl `action`. All by default. |

A call that gets no `2xx` is sent again, up to 3 times, with the same `eventId`. If your endpoint answers slowly, a `page` event can carry only the page's `url`: read that page with the status call. Signatures and payloads are in [Webhooks](https://docs.stophy.dev/webhooks.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.