> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stophy.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# RAG from websites

> Crawl a docs section, split each page into chunks that keep their source URL and heading, and load them into a search index you can cite from.

Crawl a section, split pages at headings and keep the URL on every chunk so your app can link to the source. You need `STOPHY_API_KEY`. See the [Quickstart](https://docs.stophy.dev/quickstart.md).

## 1. Crawl and read the section

Crawl follows links from the start URL and reads what it finds within `limit`. This request stops after 3 pages, enough to build the pipeline. Raise `limit` for the whole section. The `crawlResults` helper polls until the crawl ends and yields each page as it finishes.

```bash cURL theme={null}
curl -X POST https://api.stophy.dev/v1/crawl \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://bun.com/docs/runtime", "limit": 3 }'
```

```typescript TypeScript theme={null}
import { Stophy } from "stophy";
import { chunkPage } from "./chunk";

const stophy = new Stophy();

const { id } = await stophy.crawl({ url: "https://bun.com/docs/runtime", limit: 3 });
if (!id) throw new Error("The crawl did not start.");
for await (const page of stophy.crawlResults(id)) {
  const chunks = chunkPage({
    url: page.metadata?.sourceURL ?? "",
    title: page.metadata?.title ?? "",
    markdown: page.markdown ?? "",
  });
  console.log(chunks.length);
}
```

```python Python theme={null}
from stophy import Stophy

stophy = Stophy()

job = stophy.crawl(url="https://bun.com/docs/runtime", limit=3)
for page in stophy.crawl_results(job["id"]):
    print(page["metadata"]["sourceURL"], len(page["markdown"]))
```

With cURL, poll `GET /v1/crawl/{id}` and follow `next`. See [Read a crawl](https://docs.stophy.dev/crawl/async.md). Pages are kept for 24 hours. `errors` lists pages that failed. To keep a crawl to one part of a site, use `includePaths` and `excludePaths`. See [Scope and limits](https://docs.stophy.dev/crawl/overview#scope).

## 2. Cut each page into chunks

Split at headings and keep fenced code whole, so a shell comment that starts with `#` is not read as a heading. A chunk ends at the first blank line after about 1,200 characters, so a chunk can run longer. Every chunk keeps the `url`, page `title` and `heading`. Save this as `chunk.ts`.

````typescript theme={null}
export interface Page {
  url: string;
  title: string;
  markdown: string;
}

export interface Chunk {
  id: string;
  url: string;
  title: string;
  heading: string;
  text: string;
}

const MAX_CHARS = 1200;

export function chunkPage({ url, title, markdown }: Page): Chunk[] {
  const chunks: Chunk[] = [];
  let heading = title;
  let text = "";
  let inFence = false;

  const flush = () => {
    if (text.trim() !== "") {
      chunks.push({ id: `${url}#${chunks.length}`, url, title, heading, text: text.trim() });
    }
    text = "";
  };

  for (const line of markdown.split("\n")) {
    if (line.startsWith("```")) inFence = !inFence;
    if (!inFence && /^#{1,3} /.test(line)) {
      flush();
      heading = line.replace(/^#+ /, "");
    } else if (!inFence && line === "" && text.length >= MAX_CHARS) {
      flush();
    } else {
      text += `${line}\n`;
    }
  }
  flush();
  return chunks;
}
````

## 3. Index with the source

Embed `text` with your model and store the whole chunk next to the vector. The `id` is the page URL plus the chunk number. When you crawl a page again, delete the chunks with its `url` first, so none stay behind. When retrieval returns a chunk, show `title`, `heading` and a link to `url`, and put the `url` in the prompt so the model cites it.

## 4. Keep it fresh

Crawl again to rebuild the index. To re-read only what changed, create a [monitor](https://docs.stophy.dev/monitors/targets.md#crawl-targets) with a `crawl` target on the same section. It sends a `monitor.page` event for each page that changed or appeared. Read those pages with [Scrape](https://docs.stophy.dev/scrape/overview) and replace their chunks.

## Check it

* The crawl's `completed` equals its `total`, and `errors` is empty.
* Every chunk's `url` is under your start page.
* A page with 1 chunk and 20,000 characters means the headings were not split.
* Open the `url` of three chunks and find the text. If you cannot, fix the chunker before you embed.
* Ask three questions you know the answer to. The right page should come back in the top results.

See [Pricing](https://docs.stophy.dev/account/pricing.md) for crawl and monitor costs. See [Page content](https://docs.stophy.dev/crawl/overview#page-content) for what each page returns.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.