TL;DR
There is a small file you almost certainly do not have at /llms.txt, a Markdown map that hands AI crawlers a clean, curated summary of your most important content so they describe your brand correctly instead of guessing. It was proposed by Jeremy Howard of Answer.AI in September 2024, and adoption is still tiny: about 4% of the top 300 domains and ~1.2% of the top million serve one. It is also contested, Google has publicly dismissed it. So is it worth your time? The honest answer: it is cheap insurance for documentation-heavy and product sites, and a useful forcing function for thinking about how machines read you, but it is not a magic ranking lever.
The file that tells AI what you actually are
When an answer engine builds a mental model of your company, it scrapes whatever it can reach and infers the rest. If your site is a sprawl of marketing pages, blog posts, legacy docs, and JavaScript-rendered apps, the model's inference is often wrong, it conflates an old product name with a new one, misses your flagship feature, or describes you as a competitor's category. You have no clean, machine-first summary of "here is what we do and here are the canonical pages that explain it."
That gap is what llms.txt aims to fill. Proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, on 3 September 2024, it is a Markdown file at the root of your site (/llms.txt) intended for LLMs the way robots.txt is for crawlers and sitemap.xml is for search engines. Per Search Engine Land's coverage of the proposal, the idea is to give models a curated, low-noise entry point: a short description of the site, then organized links to the pages that matter most, ideally pointing at clean Markdown versions of each.
llms.txt is the curated index, an "executive summary" with structured links to your priority content. llms-full.txt is the full corpus, your entire documentation concatenated into one consumable Markdown file. The index helps a model navigate; the full file lets it ingest everything in one fetch without crawling and rendering dozens of pages.
What the format actually looks like
It is deliberately simple, plain Markdown a human can read and a model can parse without rendering. An H1 with your site or product name, an optional blockquote summary, then H2 sections grouping links with short descriptions. There is no XML, no schema, no special syntax to get wrong.
# alt.qa
> AI quality engineering platform. Three product wedges:
> Hit (browser-native LLM and API load, latency and cost testing),
> Scan (site/AEO/accessibility/Core Web Vitals/security scanning),
> Eval (AI output evaluation: hallucination, regression, RAG, bias, safety).
## Products
- [Hit, load & latency testing](https://alt.qa/lab/hit): TTFT, inter-token
latency, tokens/sec, cost per request; no account, no script.
- [Scan, site quality](https://alt.qa/lab/scan): accessibility (WCAG),
AEO crawlability, Core Web Vitals, security headers, in one pass.
- [Eval, output quality](https://alt.qa/lab/eval): continuous evaluation,
regression gates, groundedness scoring for AI outputs.
## Key guides
- [Time to First Token is your real SLA](https://alt.qa/blog/time-to-first-token-is-your-real-sla.php)
- [Invisible to AI](https://alt.qa/blog/invisible-to-ai.php)
## Optional
- [Pricing](https://alt.qa/pricing)
- [Changelog](https://alt.qa/changelog)The discipline of writing it is half the value. Forcing yourself to state, in two sentences, what your product is and which ten pages explain it, is exactly the clarity an answer engine needs, and the exercise often surfaces that your own site does not have a single clean page that says it.
The honest part: adoption is low and it is contested
I will not oversell this. Adoption is real but small. Per Frank Sauerburger's 2025 adoption analysis, the highest uptake is about 4% among the top 300 domains, falling to roughly 1.2% across the top million. A turning point came when Mintlify, a developer-docs host, rolled out automatic llms.txt support, instantly giving thousands of documentation sites (including Anthropic and Cursor) the file. So the install base skews heavily toward technical documentation.
The bigger caveat: no major LLM provider has officially committed to consuming llms.txt, and Google has publicly dismissed it, with John Mueller comparing it to the long-discredited keywords meta tag. Critics like Kai Spriestersbach have argued it is effectively a dud without provider buy-in. That critique is fair and you should weigh it.
Where it actually helps today
Be specific about where the payoff is real versus speculative. Per the practitioner experience documented in IndexLab's "does it actually work" analysis, the clearest wins are:
- Developer documentation. When a user asks an AI coding assistant "how do I authenticate with X's API, " tools that fetch
llms.txtorllms-full.txtcan pull clean, current docs instead of a rendered, ad-cluttered page. This is the strongest, most-validated use case. - Disambiguation. If your brand name collides with other entities, a concise summary that states plainly what you are gives models a canonical reference to anchor on.
- Curating signal over noise. Pointing models at your ten best pages instead of letting them wander your whole site improves the odds they cite the page you want cited.
What it does not do: it is not a ranking factor, it will not force a model that ignores the file to read it, and it does not replace the fundamentals, server-rendered content, allowed AI crawlers, valid structured data, and fresh pages. Think of llms.txt as the cherry on a properly built AEO sundae, not the sundae.
Serve it, then verify it is reachable and current
If you ship one, treat it like any other production artifact: it must be reachable, valid Markdown, and kept in sync with your real pages. A stale llms.txt pointing at dead URLs is worse than none, it actively misinforms.
# Verify your llms.txt is served, well-formed, and not rotting
import requests, re
base = 'https://example.com'
r = requests.get(f'{base}/llms.txt', timeout=10)
assert r.status_code == 200, f'llms.txt not served: {r.status_code}'
assert r.headers.get('content-type', '').startswith(('text/plain', 'text/markdown')), \
'serve as text/plain or text/markdown, not text/html'
# Every linked URL should resolve (no rot)
links = re.findall(r'\]\((https?://[^)]+)\)', r.text)
for url in links:
head = requests.head(url, allow_redirects=True, timeout=10)
if head.status_code >= 400:
print(f'DEAD link in llms.txt: {url} -> {head.status_code}')
assert r.text.lstrip().startswith('#'), 'must start with an H1 site name'
print(f'OK: {len(links)} links, {len(r.text.split())} words')The bottom line
llms.txt is a low-cost, low-risk, contested convention for handing AI a clean map of your content. Adoption is still small (~1.2% of the top million), no major provider has formally committed to it, and Google has dismissed it, so do not expect it to move rankings. But for documentation and product sites it is a genuine, validated win, and for everyone it is near-free disambiguation insurance and a useful exercise in stating, machine-first, what you actually are. Ship one, keep it current, verify it is reachable, and treat it as the finishing touch on the AEO fundamentals, server-side rendering, crawler access, structured data, and freshness, that actually carry the weight.
Find the Gaps Before They Cost You
Scan audits your site for the accessibility, performance, AEO, and security gaps that quietly drain revenue and invite lawsuits, in one pass.
Try Scan Free →