AI crawler access and llms.txt: useful foundations, not a strategy
Most of what gets said about AI crawler access and llms.txt overstates what either one actually does. Blocking AI crawlers by accident is a real and common problem, worth checking. Adding llms.txt is a sensible, low-cost habit. Neither is a visibility strategy by itself.
What actually controls AI crawler access
Your robots.txt file is the real gatekeeper. It’s worth checking that a blanket “block all bots” rule — often added for a legitimate reason, like keeping out scrapers — hasn’t accidentally caught the crawlers you want in:
- GPTBot and OAI-SearchBot (OpenAI)
- PerplexityBot (Perplexity)
- ClaudeBot (Anthropic)
- Google-Extended (Google’s AI training and Gemini use, separate from the standard Googlebot rule)
If any of these are disallowed and you didn’t do it deliberately, that’s the first thing to fix — before anything else on this list.
What llms.txt actually is
llms.txt is a plain-text file at your site’s root (/llms.txt) that summarises who you are, what you do and links to your key pages, aimed at AI tools rather than search engines. It’s a young, informal convention — not a ranking signal, not a guaranteed citation mechanism, and not adopted uniformly across platforms. Chapman’s Consulting’s own llms.txt is a working example: an entity summary, service links, proof and a contact route, kept in sync with the site.
Worth having. Not worth treating as a strategy on its own.
What it can’t do
Be sceptical of anyone selling llms.txt as a fix for weak visibility:
- No platform has confirmed it affects what gets cited or recommended.
- It doesn’t substitute for a crawlable, well-structured site — it’s a summary of content that needs to already be good.
- It can’t fix inaccurate or inconsistent information elsewhere about your business.
A practical checklist
- Check
robots.txtfor accidental disallow rules against GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended. - Confirm your key pages’ main content renders without requiring JavaScript, since not every crawler executes scripts.
- Add or update an
llms.txtfile: a plain entity summary, your core service links, and proof — kept current as the site changes. - Make sure your business name, description and services read the same way across your site, LinkedIn and any directories — consistency helps every AI tool that cross-checks sources, with or without llms.txt.
- Treat all of the above as foundations, then put the real effort into answer-first content and quotable, specific facts — what actually moves AI visibility →.
Foundations like these are exactly what’s checked in an AI visibility audit, alongside the content and entity work that does more of the heavy lifting. More about AI visibility and GEO →
FAQs
What is llms.txt?
llms.txt is a plain-text file at the root of a website (chapmansconsulting.co.uk/llms.txt, for example) that summarises who you are and links to your key pages, written for AI tools rather than search crawlers. It’s a young, informal convention, not a ranking or citation mechanism.
Does llms.txt help you rank in ChatGPT?
No platform has confirmed that llms.txt affects what it cites or recommends. It’s a helpful, low-cost summary to have in place, not something with a proven causal effect on AI answers. Treat it as good housekeeping, not a strategy.
How do I check if AI crawlers can access my site?
Check your robots.txt for disallow rules that block bots such as GPTBot, OAI-SearchBot, PerplexityBot or ClaudeBot — sometimes added by accident as part of a broader 'block all bots’ rule. Then confirm your key pages render without needing JavaScript to show their main content, since some crawlers don’t execute scripts.
What’s the difference between robots.txt and llms.txt?
robots.txt tells crawlers what they’re allowed to access — it controls permission. llms.txt is a voluntary summary for AI tools that choose to read it — it doesn’t control access at all. Both are worth having; neither replaces the other.
The Point, once a month
A short note on what’s worth doing in marketing operations, SEO and AI visibility, and what to stop.
Let’s talk
A 30-minute discovery call is free and there’s no pitch. Bring the situation as it is. If I’m not the right fit, I’ll say so, and suggest who might be.