STOPHY_API_KEY. See the Quickstart.
1. Crawl and read the section
Crawl follows links from the start URL and reads what it finds withinlimit. This request stops after 3 pages, enough to build the pipeline. Raise limit for the whole section. The crawlResults helper polls until the crawl ends and yields each page as it finishes.
GET /v1/crawl/{id} and follow next. See Read a crawl. Pages are kept for 24 hours. errors lists pages that failed. To keep a crawl to one part of a site, use includePaths and excludePaths. See Scope and limits.
2. Cut each page into chunks
Split at headings and keep fenced code whole, so a shell comment that starts with# is not read as a heading. A chunk ends at the first blank line after about 1,200 characters, so a chunk can run longer. Every chunk keeps the url, page title and heading. Save this as chunk.ts.
3. Index with the source
Embedtext with your model and store the whole chunk next to the vector. The id is the page URL plus the chunk number. When you crawl a page again, delete the chunks with its url first, so none stay behind. When retrieval returns a chunk, show title, heading and a link to url, and put the url in the prompt so the model cites it.
4. Keep it fresh
Crawl again to rebuild the index. To re-read only what changed, create a monitor with acrawl target on the same section. It sends a monitor.page event for each page that changed or appeared. Read those pages with Scrape and replace their chunks.
Check it
- The crawl’s
completedequals itstotal, anderrorsis empty. - Every chunk’s
urlis under your start page. - A page with 1 chunk and 20,000 characters means the headings were not split.
- Open the
urlof three chunks and find the text. If you cannot, fix the chunker before you embed. - Ask three questions you know the answer to. The right page should come back in the top results.