> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Batches

> Read a list of up to 10,000 URLs in the background with scrape options shared by every page. Submit once, get an id, then read pages as they finish.

A batch reads a list of URLs in the background, with the same scrape options for every page.

```bash cURL theme={null}
curl -X POST https://api.stophy.dev/v1/batch/scrape \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com", "ftp://example.com/file.txt"]
  }'
```

```typescript TypeScript theme={null}
import { Stophy } from "stophy";

const stophy = new Stophy();

const batch = await stophy.batch.scrape({
  urls: ["https://example.com", "ftp://example.com/file.txt"],
});
console.log(batch.id, batch.invalidURLs);
```

```python Python theme={null}
from stophy import Stophy

stophy = Stophy()

batch = stophy.batch.scrape(urls=["https://example.com", "ftp://example.com/file.txt"])
print(batch["id"], batch["invalidURLs"])
```

Send 1 to 10,000 URLs of at most 2,048 characters each. Each page comes back as Markdown, and an address sent twice is read once. Batches need an API key, see [Get credentials](https://docs.stophy.dev/get-credentials.md).

```json theme={null}
{
  "success": true,
  "id": "job_wfE0hfue0Xn7yNS-9kq-Ng",
  "url": "https://api.stophy.dev/v1/batch/scrape/job_wfE0hfue0Xn7yNS-9kq-Ng",
  "invalidURLs": ["ftp://example.com/file.txt"]
}
```

The call answers at once. `invalidURLs` lists the addresses that are not public website URLs: they are skipped and the rest runs. A batch that waits for a free place also has `status` and `queuePosition`. Read the pages from `url` as they finish, see [Batch results](https://docs.stophy.dev/batches/results.md). If you want the URLs found for you, use [Crawl](https://docs.stophy.dev/crawl/overview).

## Fields

| Field | Default | What it does |
| - | - | - |
| Any [Scrape](https://docs.stophy.dev/scrape/overview) field but `url` and `offset` | | Applies to every page, such as `"formats": ["markdown", "links"]`. |
| `ignoreInvalidURLs` | `true` | `false` refuses the whole batch when one address is invalid. |
| `maxConcurrency` | none | Pages read at once, 1 to 100. A batch reads at most 4 at once. |
| `webhook` | none | Be told as each page finishes and at the end. See [Webhooks](https://docs.stophy.dev/webhooks.md). |
| `appendToId` | none | Add `urls` to a running batch, see below. |

The [API reference](https://docs.stophy.dev/api-reference/endpoint/batch-scrape.md) has every field.

## Add URLs to a running batch

Send the batch `id` in `appendToId` and the new addresses in `urls`. The batch keeps its own options and webhook, skips URLs it already has, and counts the new ones in `total`. Added URLs count toward the 10,000 cap.

## Retry a submit safely

Send an `Idempotency-Key` header with any text. The same key and body within 24 hours return the first answer with `idempotent-replayed: true`. The same key with a different body starts a new batch.

## Errors

| Code | Cause | Fix |
| - | - | - |
| `400 invalidRequest` | No public URL in `urls`, more than 10,000, a bad `webhook`, or an invalid address with `ignoreInvalidURLs: false` | Fix the field that `error` names |
| `409 jobFinished` | `appendToId` names a batch that ended | Start a new batch |
| `429 rateLimited` | More jobs running or waiting than your tier allows | Wait `retryAfterSeconds`. See [Rate limits](https://docs.stophy.dev/production/rate-limits.md). |

A batch ends with `status: "failed"` when your balance runs out or no page could be read. `completed` means every page was tried, so check `errors`. Every code is in [Errors](https://docs.stophy.dev/api-reference/errors.md).

## Next

* [Batch results](https://docs.stophy.dev/batches/results.md): read, retry, cancel and list batches.
* [Scrape](https://docs.stophy.dev/scrape/overview): the options every page takes.
* [Map](https://docs.stophy.dev/map/overview): list a site's URLs to send.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.