Turn any URL into clean Markdown for an LLM

Raw HTML is a poor thing to give a language model. A typical article page is mostly navigation, scripts, tracking tags and footers: the words you want are a small part of it, and the rest fills the context window. Readmint's read operation returns only the main content, as Markdown.

One call

curl https://api-readmint.plxi.us/v1/read \
  -H "X-API-Key: $READMINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://en.wikipedia.org/wiki/Tide_pool"}'

The answer has the page's title, byline, published date and lang, and content: the article as Markdown, with headings, lists, tables and code blocks kept and every link and image made a full URL.

{
  "url": "https://en.wikipedia.org/wiki/Tide_pool",
  "final_url": "https://en.wikipedia.org/wiki/Tide_pool",
  "status": 200,
  "title": "Tide pool",
  "format": "markdown",
  "content": "A **tide pool** or **rock pool** is a shallow pool of [seawater](https://en.wikipedia.org/wiki/Seawater) …",
  "word_count": 2895,
  "truncated": false,
  "rendered": false,
  "credits": 1
}

What gets left out

Mozilla's Readability, the library behind Firefox's Reader View, finds the main content. Navigation, headers, footers, sidebars, cookie banners, pop-ups and chat widgets are dropped. A page that isn't an article, such as a home page or a pricing page, comes back without its navigation, header and footer instead.

When you know where the content is, read just that part with a CSS selector:

{"url": "https://docs.python.org/3/library/json.html", "selector": "section#module-json"}

Pages built by JavaScript

Readmint fetches each page directly first, which takes a fraction of a second. When that finds almost no text, a page that asks for JavaScript, or a site that turns plain clients away, it loads the page in a real Chromium browser instead. rendered in the answer says which happened. A read costs one page either way. render: "always" or "never" overrides the choice.

Keep it inside the context window

max_chars caps the content (100,000 characters by default); past it the content is cut and truncated is true. images: false drops image links, which are mostly noise to a text model. format: "text" gives plain text with paragraph breaks if you don't want Markdown at all.

From code

import os, requests

page = requests.post(
    "https://api-readmint.plxi.us/v1/read",
    headers={"X-API-Key": os.environ["READMINT_KEY"]},
    json={"url": "https://blog.rust-lang.org/", "max_chars": 20000, "images": False},
).json()

prompt = f"Summarise this page:\n\n# {page['title']}\n\n{page['content']}"

const res = await fetch('https://api-readmint.plxi.us/v1/read', {
  method: 'POST',
  headers: { 'X-API-Key': process.env.READMINT_KEY, 'Content-Type': 'application/json' },
  body: JSON.stringify({ url: 'https://blog.rust-lang.org/', max_chars: 20000 }),
});
const { title, content } = await res.json();

When a page can't be read

Readmint reads public pages. It refuses private addresses, pages behind a bot check, a CAPTCHA or a login, HTTP errors and files that aren't web pages, with a 4xx error that says why. None of those are charged. See the API reference for every option.