Skip to main content
Crawl a section, split pages at headings and keep the URL on every chunk so your app can link to the source. You need STOPHY_API_KEY. See the Quickstart.

1. Crawl and read the section

Crawl follows links from the start URL and reads what it finds within limit. This request stops after 3 pages, enough to build the pipeline. Raise limit for the whole section. The crawlResults helper polls until the crawl ends and yields each page as it finishes.
With cURL, poll GET /v1/crawl/{id} and follow next. See Read a crawl. Pages are kept for 24 hours. errors lists pages that failed. To keep a crawl to one part of a site, use includePaths and excludePaths. See Scope and limits.

2. Cut each page into chunks

Split at headings and keep fenced code whole, so a shell comment that starts with # is not read as a heading. A chunk ends at the first blank line after about 1,200 characters, so a chunk can run longer. Every chunk keeps the url, page title and heading. Save this as chunk.ts.

3. Index with the source

Embed text with your model and store the whole chunk next to the vector. The id is the page URL plus the chunk number. When you crawl a page again, delete the chunks with its url first, so none stay behind. When retrieval returns a chunk, show title, heading and a link to url, and put the url in the prompt so the model cites it.

4. Keep it fresh

Crawl again to rebuild the index. To re-read only what changed, create a monitor with a crawl target on the same section. It sends a monitor.page event for each page that changed or appeared. Read those pages with Scrape and replace their chunks.

Check it

  • The crawl’s completed equals its total, and errors is empty.
  • Every chunk’s url is under your start page.
  • A page with 1 chunk and 20,000 characters means the headings were not split.
  • Open the url of three chunks and find the text. If you cannot, fix the chunker before you embed.
  • Ask three questions you know the answer to. The right page should come back in the top results.
See Pricing for crawl and monitor costs. See Page content for what each page returns.