Web pages for RAG: clean Markdown to chunk and embed

Retrieval-augmented generation is only as good as the text you index. Pages scraped as raw HTML or naive text bring navigation menus, cookie notices and "related articles" into every chunk, and those chunks then match queries they have nothing to do with.

What you get from each page

POST /v1/read returns:

• content: the main content as Markdown. Headings stay headings, so you can split on them; tables stay tables; links and images are full URLs.
• title, byline, published, lang, site_name: metadata to store next to each chunk and to filter on.
• final_url, and with include: ["metadata"] the page's canonical URL: the right keys for de-duplicating.
• word_count and truncated, so you can spot pages that are mostly empty or were cut at max_chars.

A pipeline

1. Find the pages with POST /v1/map: the site's sitemaps plus the links on a start page, same site only. Filter with search (/docs/, /help/).
2. Read each page with POST /v1/read, images: false, a few at a time.
3. Split the Markdown on headings, then by length.
4. Embed each chunk with its title, URL and date as metadata.
5. Refresh on a schedule: send max_age: 0 to skip the cache, and compare content with what you stored to re-embed only what changed.

The guide to reading a whole site has the code for steps 1 and 2.

Pages built by JavaScript

Single-page apps and some docs sites send almost no HTML. Readmint notices and loads those pages in a real browser, so your pipeline doesn't need its own headless Chrome. A read costs one page whether or not it needed the browser.

What it won't do

Readmint reads public pages. It doesn't log in, solve CAPTCHAs or get past bot protection, and doesn't read PDFs yet. Pages it can't read are refused with a 4xx error and aren't charged.