Read a whole website as Markdown, page by page

To put a documentation site or a blog into a search index or a model's knowledge, you need two things: the list of its pages, and each page as clean text. Readmint's map gives the first and read the second.

1. List the pages

curl https://api-readmint.plxi.us/v1/map \
  -H "X-API-Key: $READMINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://blog.rust-lang.org/", "search": "2024", "max_urls": 500}'

map reads the sitemaps the site lists in robots.txt (or /sitemap.xml), including sitemap indexes and .gz files, and the links on the start page. It keeps only pages on the same site, without duplicates or #fragments. search keeps URLs containing some text, such as /docs/ or a year; include_subdomains: true keeps docs.example.com for example.com. A map costs one page.

{
  "url": "https://blog.rust-lang.org/",
  "urls": ["https://blog.rust-lang.org/", "https://blog.rust-lang.org/2024/02/08/Rust-1.76.0/", "…"],
  "count": 120,
  "from_sitemap": 119,
  "from_page": 0,
  "sitemaps": ["https://blog.rust-lang.org/sitemap.xml"],
  "truncated": false,
  "credits": 1
}

2. Read each one

Read the pages you chose, a few at a time. Each read is one page; a page that fails (a 404, a bot check) is refused with a 4xx error and isn't charged, so skip it and carry on.

import os, requests, concurrent.futures as cf

API = "https://api-readmint.plxi.us"
HEADERS = {"X-API-Key": os.environ["READMINT_KEY"]}

urls = requests.post(f"{API}/v1/map", headers=HEADERS,
                     json={"url": "https://blog.rust-lang.org/", "search": "2024"}).json()["urls"]

def read(url):
    r = requests.post(f"{API}/v1/read", headers=HEADERS, json={"url": url, "images": False})
    return r.json() if r.ok else None

# Stay at or under your plan's pages-at-a-time limit (2 on Free, 4 on Starter).
with cf.ThreadPoolExecutor(max_workers=2) as pool:
    pages = [p for p in pool.map(read, urls) if p]

for p in pages:
    print(p["title"], p["word_count"], p["final_url"])

Tips

• Store final_url, not the URL you asked for: it's where the page ended up after redirects, and the right key for de-duplicating.
• published and byline come from the page when it gives them; keep them as metadata next to each chunk.
• A 429 concurrency_limit_exceeded means you sent more pages at once than your plan allows. Retry it after a short wait.
• Readmint has no crawl operation yet: map then read is the way to cover a site, and it lets you skip pages you don't need.