Does blocking GPTBot remove me from ChatGPT?

Blocking GPTBot stops OpenAI from using your pages for training. ChatGPT search uses OAI-SearchBot and user-triggered fetches use ChatGPT-User, so block or allow those separately.

Is llms.txt required?

No. It is a voluntary convention. It is quick to add and gives AI tools a clean summary of your site, but crawler access, clear page text and third-party mentions matter more.

llms.txt, robots.txt and AI crawlers: how to make your website readable by AI

By Get AI Vizible · Updated 24 October 2026

Short answer

To make your site readable by AI assistants, allow the AI crawlers in robots.txt (OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic's ClaudeBot, Claude-SearchBot and Claude-User; PerplexityBot and Perplexity-User; Google-Extended; Applebot-Extended), serve your key facts as HTML text, add schema markup, and publish an llms.txt file: a short Markdown summary of who you are with links to your most important pages.

AI crawlers to know

Company	User agent	Used for
OpenAI	GPTBot	Training data
OpenAI	OAI-SearchBot	ChatGPT search results
OpenAI	ChatGPT-User	Pages fetched when a user asks
Anthropic	ClaudeBot	Training data
Anthropic	Claude-SearchBot, Claude-User	Search and user-requested fetches
Perplexity	PerplexityBot, Perplexity-User	Search index and user-requested fetches
Google	Google-Extended	Use in Gemini (a control token; crawling is done by Googlebot)
Apple	Applebot-Extended	Use in Apple's AI features
Common Crawl	CCBot	Open dataset used by many models

Blocking a search or user agent removes you from that engine's live answers. Blocking a training agent only affects future model versions. Decide per agent; for most brands that want to be recommended, allowing all of them is the simplest choice.

A robots.txt that allows AI crawlers

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

What llms.txt is

llms.txt is a proposed convention: a Markdown file at /llms.txt that gives language models a short, clean summary of your site. It typically has:

a title and one-paragraph description
key facts (what you do, who it is for, pricing, contact)
links to the most useful pages with one line each

Some sites also publish llms-full.txt with the full text of their key pages. Support differs by engine, so treat llms.txt as a cheap, helpful extra rather than a guarantee. You can see ours at /llms.txt.

Other technical checks

Server-rendered text. Prices, features and FAQs should be in the HTML, not loaded only by JavaScript.
Schema markup. Organization, Product or SoftwareApplication, FAQPage, Article and BreadcrumbList, matching the visible text.
A sitemap listing every public page with accurate last-modified dates.
Fast, stable pages without interstitials that hide content.
One canonical URL per page.

Get AI Vizible's site check tests AI crawler access, llms.txt, schema and crawlable text on your site and tells you what to fix.

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT?: Blocking GPTBot stops OpenAI from using your pages for training. ChatGPT search uses OAI-SearchBot and user-triggered fetches use ChatGPT-User, so block or allow those separately.
Is llms.txt required?: No. It is a voluntary convention. It is quick to add and gives AI tools a clean summary of your site, but crawler access, clear page text and third-party mentions matter more.