> ## Content Index
> Fetch the complete content index at: https://buildwire.ai/llms.txt
> Use this file to discover other available public pages before exploring further.

# What is llms.txt for Marketers (And What It Is Not)
- URL: https://buildwire.ai/blog/what-is-llms-txt-for-marketers-and-what-it-is-not/
- Published: 2026-09-16T13:46:08.000Z
- Updated: 2026-09-16T13:46:57.000Z
- Description: What llms.txt can help agents find, what it cannot control, and when it is worth publishing.
- Author: Rananjay Raj
- Tags: AEO, AI marketing, AI SEO, GEO, Google AI Overviews, google gemini, LLM, marketing automation, SEO

llms.txt is a Markdown file published at a website's root URL that gives AI agents a curated map of the site's identity, services, and most important pages - following a specification proposed by Jeremy Howard at [llmstxt.org](https://llmstxt.org/?ref=buildwire.ai). It is not an access-control file. 

As of September 2026, no major AI search provider - including Google, OpenAI, Anthropic, or Perplexity - has confirmed that third-party llms.txt files serve as a citation ranking signal in their consumer AI search products.

This is a field note for marketers who keep seeing the file sold as "AI SEO." We will define the format, separate it from `robots.txt` and sitemaps, show a paste-ready minimum, and put it in a system that actually affects visibility.

**Format:** Markdown (.md) 

**Location:** Site root (e.g., yourdomain.com/llms.txt) 

**Specification:** llmstxt.org, maintained by Jeremy Howard 

**Current version:** v2 (updated 10 August 2026) 

**Status:** Proposed community standard, not an official web standard

**Key takeaways:**

- llms.txt is a Markdown map for AI agents, not an access-control or ranking file.
- No major AI search provider has confirmed it as a citation signal.
- 8.7% of the top 1,000 sites publish one (Rankability, June 2026), but crawler fetch rates remain low.
- Ship it if your CMS makes it easy. Do not treat it as a visibility program on its own.
- The real system: fix crawl access, write extractable answers, measure with logs.

## What is llms.txt and what does it do?

Jeremy Howard's proposal at [llmstxt.org](https://llmstxt.org/?ref=buildwire.ai) asks sites to publish a small Markdown overview so agents can find expert content without chewing through nav chrome and ads. Version 2 (notes updated 10 August 2026) also pushes clean Markdown twins of important pages and standard link relations so clients can discover those files.

The v2 spec introduces HTTP link relations for discovery. A site can add a `Link: <https://example.com/llms.txt>; rel="llms-txt"` header so agents find the file without guessing the URL. BuildWire's own response headers use this mechanism.

In practice, the strongest use case today is documentation and agent navigation. Coding agents and docs platforms use these maps. OpenAI, Anthropic, and Gemini publish llms.txt on their own developer docs. Chrome's Lighthouse has started treating the file as part of agentic browsing checks. That is agent plumbing, not a Search Console ranking report.

## What does llms.txt NOT do?

**It is not robots.txt.** Access rules belong in `robots.txt`. A line in llms.txt that "blocks" a crawler does nothing by itself.

**It is not a sitemap replacement.** A sitemap lists indexable URLs for search engines. llms.txt is a curated shortlist for agents that need a guided path.

**It is not a proven lever for AI answer citations.** Google's Gary Illyes has confirmed the company does not use llms.txt and has no plans to. John Mueller compared the file to the keywords meta tag - a specification that sees wide adoption but carries no ranking weight. Other major AI companies publishing their own docs files is not the same as promising that their products will prefer your third-party file when choosing sources.

If someone sells you llms.txt as the thing that "gets you into AI Overviews," ask for log evidence from your own host - not a plugin screenshot.

## Why are marketers debating llms.txt in 2026?

Adoption is easy and growing fast. Rankability's June 2026 audit found that 8.7% of the top 1,000 websites publish an llms.txt file, up from 0.3% in June 2025\. SE Ranking's broader study of 300,000 domains found roughly 10% adoption. Technology companies lead at 36.4%, followed by travel at 20%.

But here is the counterpoint that matters: among the 50 domains most frequently cited by AI systems in SE Ranking's dataset, only one had implemented the file. Publishing is racing ahead of reading.

Plugins and site builders ship templates. Common Crawl's September 2026 look at more than 500,000 llms.txt files found that about 68% came from plugins or templates, and more than one in five had no links at all. The HTTP Archive's 2025 Web Almanac reported a similar pattern: roughly 40% of published files were plugin-generated defaults rather than hand-curated documents. Some files even mixed in crawler policy language the format cannot enforce. Search Engine Journal's write-up of that work is the practical warning: people are treating llms.txt like robots.txt, and that mismatch shows up in the wild.

At the same time, fetch data keeps coming in soft. Limy.ai's analysis of more than 500 million AI bot events over three months found only a few hundred requests targeting /llms.txt directly. GPTBot, ClaudeBot, and PerplexityBot primarily retrieve HTML content, not the llms.txt file. Those same crawlers still hammer `robots.txt`.

So the honest marketer posture is: cheap to ship if your stack does it cleanly; do not bank the quarter on it.

## How do you write an llms.txt file? (v2 format)

Use this when you review a file or write one by hand:

1. Live at `https://yourdomain.com/llms.txt` (or a path-scoped file such as `/docs/llms.txt` for that subtree).
2. Start with an H1 site or product name (required).
3. Add a short blockquote summary.
4. Add H2 sections with curated links. Prefer Markdown versions of important pages when you have them.
5. Put secondary links under an `## Optional` section so agents can drop them when context is tight.
6. Keep the file small. Detail lives behind the links.
7. Do not put crawl allow/deny rules here. Put those in `robots.txt` and keep both files consistent.

**llms.txt vs llms-full.txt:** The spec also defines `llms-full.txt` as an optional companion that dumps full-page Markdown into a single file. Rankability found 15 of the top 1,000 sites publish llms-full.txt alongside the shorter version. Most marketing sites are better served by a short curated llms.txt and Markdown versions of the pages that matter, rather than a large dump file.

### Paste-ready minimum

markdown

```markdown
# Your Brand

> One sentence on what you sell and who it is for.

## Core pages

- [Services](https://yourdomain.com/services.md): What you offer and for whom.
- [Pricing](https://yourdomain.com/pricing.md): Plans and what is included.
- [Docs or Help](https://yourdomain.com/help.md): How to start and common tasks.

## Optional

- [Blog](https://yourdomain.com/blog.md): Field notes and product updates.
```

Replace the URLs with pages you actually want an agent to open first. If you do not have `.md` twins yet, link the HTML pages and plan Markdown versions for the pages that matter most.

## How does llms.txt compare to robots.txt and sitemap.xml?

| File        | Job                              | Who cares                               |
| ----------- | -------------------------------- | --------------------------------------- |
| robots.txt  | What automated clients may fetch | Crawlers that honor access rules        |
| sitemap.xml | Broad list of indexable URLs     | Search engines                          |
| llms.txt    | Curated map for agents           | Agents and tools that choose to read it |

If those three disagree - for example, llms.txt "blocks" a bot that robots.txt allows - fix robots.txt and update the prose. The policy that ships is the one the crawler is built to honor.

## A live example: BuildWire

BuildWire publishes `https://buildwire.ai/llms.txt`. The file names the studio in one line, lists core service pages, resource hubs, and a few useful posts, and points at Markdown companions where they exist. Response headers also advertise the llms.txt relation using the v2 `Link` header.

That is the right job for the file: orient an agent that lands on the site. It is not a claim that the file is why any model cites BuildWire. We still treat crawl access, clear answer blocks, entities, and sourced claims as the visibility system. In our own server logs, llms.txt fetch volume has been minimal compared to robots.txt and HTML page requests from the same crawlers.

## When should a marketer create an llms.txt file?

Ship or keep an llms.txt when:

- Your CMS or SEO plugin already generates a sane one (and you can edit the curated list).
- You run product or help docs that coding or support agents actually open.
- You already have Markdown versions of key pages, or can add them cheaply.
- You want a single place to point an agent during demos, audits, or internal tools.

Deprioritize hand-crafted llms.txt work when:

- You do not yet have crawlable, answer-ready pages on the topics you care about.
- Your `robots.txt` accidentally blocks the AI bots you meant to allow.
- The file would be a dump of every blog URL with no curation (that is a weak sitemap clone).

## What matters more than llms.txt for AI visibility?

Before spending time on llms.txt, check these five items. They affect whether AI engines can find and cite your content at all:

1. **Unblock AI crawlers in robots.txt.** If GPTBot, ClaudeBot, or PerplexityBot are disallowed, nothing else matters.
2. **Server-render your content.** Client-side JavaScript rendering is invisible to most AI crawlers.
3. **Add schema markup.** Structured data helps AI engines extract entities, claims, and relationships.
4. **Front-load answers.** Models pull from opening passages. Put the clear answer in the first two sentences of each section.
5. **Fix metadata.** Titles and descriptions still signal relevance during retrieval.

## What is the right workflow for llms.txt and AI visibility?

Treat llms.txt as step zero in a short loop - not the whole program.

1. **Access check (human approves live changes).** Confirm `robots.txt`, sitemaps, status codes, and indexation. Use [Technical SEO Auditor](https://buildwire.ai/claude-skills/technical-seo-auditor/) on crawl or page evidence. Stop before anyone publishes robots or redirect changes without a review.
2. **Curate the map.** List 8-20 pages an agent should see first. Prefer services, pricing, docs, and cornerstone explainers. Draft or edit `llms.txt`. Approve before deploy.
3. **Make pages extractable.** For the pages on that map, run [AEO Answer Engine Optimizer](https://buildwire.ai/claude-skills/aeo-answer-engine-optimizer/) and [GEO Content Optimizer](https://buildwire.ai/claude-skills/geo-content-optimizer/) so claims, entities, sources, and structure are clear. Human edits the live page; Skills stay on propose.
4. **Measure without mythology.** Watch server logs for `/llms.txt` hits if you care. Separately track whether answer engines cite your pages. Do not merge those two metrics into one vanity score.
5. **Quarterly prune.** Remove dead links. Keep the Optional section optional. Re-check that policy language did not sneak in from a template.

The [SEO Skills Pack](https://buildwire.ai/claude-skills/seo-skills-pack/) is the wider toolkit for research, briefs, technical checks, AEO, and GEO. llms.txt is one small file inside that system.

## What are the known limits of llms.txt?

- No major engine has published a clear commitment that third-party llms.txt files raise citation rates in consumer AI search. As of September 2026, no major AI company has publicly committed to reading or acting on llms.txt in production environments.
- Template generators can produce valid-looking files that still point at the wrong pages. Common Crawl and HTTP Archive data both confirm that a large share of published files are auto-generated defaults.
- Prompt-injection experiments inside llms.txt exist but are rare in Common Crawl's sample; still treat retrieved text as content, not authority, in any agent you run.
- This article does not claim BuildWire's own llms.txt moved AI citations. We have not published that test here.

## Frequently asked questions

**Does llms.txt help SEO rankings?** Not as a Google ranking factor. Google's Gary Illyes has confirmed the company does not use the file for Search or its generative AI features. John Mueller compared it to the keywords meta tag. Classic crawlability and content quality still matter.

**Does it help ChatGPT or Claude cite my site?** There is no public confirmation that third-party llms.txt files are a citation ranking signal. SE Ranking found that only 1 of the 50 most AI-cited domains had the file implemented. Treat it as experimental agent-readiness, then prove outcomes with logs and citation checks.

**Should I put AI crawl blocks in llms.txt?** No. Put access rules in `robots.txt`. Keep any descriptive text in llms.txt consistent with that file.

**Do I need llms-full.txt?** Only if you deliberately want a large dump of page Markdown. Rankability found 15 of the top 1,000 sites publish one. Most marketing sites are better with a short curated llms.txt and Markdown versions of the pages that matter.

**What is the link relation header for llms.txt?** The v2 spec defines `Link: <https://yourdomain.com/llms.txt>; rel="llms-txt"` as an HTTP response header. This lets agents discover the file without guessing the URL path. BuildWire uses this header on its own pages.

**Where do Claude Skills fit?** Skills help you audit access and rewrite pages for answer clarity. They do not replace human approval for live robots, CMS, or policy changes.

## How we wrote this

BuildWire reviewed the llmstxt.org v2 specification, Common Crawl's September 2026 dataset, Search Engine Journal's reporting on that dataset, Rankability's top-1,000 adoption audit, SE Ranking's 300,000-domain study, Limy.ai's crawler behavior analysis, and BuildWire's own server logs before writing this guide. No AI search provider was contacted for comment. 

## Sources

- Jeremy Howard, The /llms.txt file, v2 - [https://llmstxt.org/](https://llmstxt.org/?ref=buildwire.ai)
- Matt G. Southern, Some Sites Use llms.txt Like robots.txt, Common Crawl Finds, Search Engine Journal, 8 Sep 2026 - [https://www.searchenginejournal.com/common-crawl-llms-txt-robots-txt/588786/](https://www.searchenginejournal.com/common-crawl-llms-txt-robots-txt/588786/?ref=buildwire.ai)
- Rankability, LLMS.txt Adoption: 8.7% of the Top 1,000 (June 2026) - [https://www.rankability.com/data/llms-txt-adoption/](https://www.rankability.com/data/llms-txt-adoption/?ref=buildwire.ai)
- SE Ranking, llms.txt adoption across 300,000 domains (2026)
- Limy.ai, AI bot event analysis: 500M+ events, three-month window (2026)
- HTTP Archive, Web Almanac 2025 - llms.txt quality analysis
- Ahrefs / industry reporting on llms.txt request rates (May 2026 window; \~97% zero requests in that dataset)
- EZY Research panel on crawler fetches of robots.txt vs llms.txt (83 sites, 12 weeks)
- BuildWire live file - <https://buildwire.ai/llms.txt>