> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Structured datasets

> Turn a list of URLs into rows with a JSON schema. Submit a batch, read one object per page, handle missing fields and write a CSV.

Use a batch to read many pages in the background with one schema, then turn each page's fields into a CSV row. You need `STOPHY_API_KEY`. See the [Quickstart](https://docs.stophy.dev/quickstart.md).

## 1. Submit the batch

The `json` format takes a `schema`. Use `required` to ask for fields on every page, and `description` to fix a format. See [JSON](https://docs.stophy.dev/scrape/json.md). The call answers at once with an `id` and a `url` to poll.

```bash theme={null}
curl -X POST https://api.stophy.dev/v1/batch/scrape \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://bun.com/blog/bun-v1.4.4",
      "https://bun.com/blog/bun-in-rust",
      "https://bun.com/blog/behind-the-scenes-of-bun-install"
    ],
    "formats": [{
      "type": "json",
      "schema": {
        "type": "object",
        "properties": {
          "title": { "type": "string" },
          "author": { "type": "string" },
          "publishedOn": { "type": "string", "description": "The publication date as YYYY-MM-DD." },
          "topics": { "type": "array", "items": { "type": "string" } }
        },
        "required": ["title", "publishedOn"]
      }
    }]
  }'
```

`invalidURLs` lists addresses that are not public pages. They are left out.

## 2. Read the answers

Read `url` until `status` is `completed`. Each page in `data` has its object in `json` and its address in `metadata.url`. This row has no `topics`: the field was optional and the page lists none.

```json theme={null}
{
  "status": "completed",
  "data": [{
    "json": { "title": "Behind The Scenes of Bun Install", "author": "example_user", "publishedOn": "2025-09-10" },
    "metadata": { "url": "https://bun.com/blog/behind-the-scenes-of-bun-install" }
  }]
}
```

A long batch comes back in parts. Follow `next` until it is missing, and read failed pages from `errors`. See [Batch results](https://docs.stophy.dev/batches/results.md).

## 3. Write the rows

Flatten each `json` object into a row. Fill a missing field with an empty string, or skip the row if your app needs it. Join `topics` with `; `.

```typescript theme={null}
export function toCsv(rows: Record<string, string>[]): string {
  const first = rows[0];
  if (first === undefined) return "";
  const columns = Object.keys(first);
  const cell = (value: string) => `"${value.replaceAll('"', '""')}"`;
  const lines = rows.map((row) => columns.map((name) => cell(row[name] ?? "")).join(","));
  return [columns.join(","), ...lines].join("\n");
}
```

## Check it

* `completed` plus `failed` equals `total`, and your row count equals the pages that returned `json`.
* Open three pages and compare each cell. The model fills fields from page text, so a field the page does not state can come back wrong or empty, even when it is `required`.
* Dates share one format. If not, tighten the `description`.
* Store `url` and the read time with each row, so you can tell how old it is.

Test the schema on 3 to 5 pages before you send the list. See [Pricing](https://docs.stophy.dev/account/pricing.md). To fill one row from several pages, use [Extract](https://docs.stophy.dev/extract/overview).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.