How to make your Next.js site appear in ChatGPT (and any LLM) You blocked GPTBot in robots.txt to keep your content out of training runs — and then wondered why ChatGPT Search never cites your docs.
By Coderz Club · 2026-08-03 · Tags: ai, html
How to make your Next.js site appear in ChatGPT (and any LLM)
You blocked GPTBot in robots.txt to keep your content out of training runs — and then wondered why ChatGPT Search never cites your docs. Those are different systems. OpenAI’s own crawler docs say each bot is independent: allowing OAI-SearchBot keeps you eligible for ChatGPT search answers while disallowing GPTBot opts you out of foundation-model training. This article is a Next.js App Router playbook for Generative Engine Optimization (GEO): how answer engines discover pages, which user-agents actually matter, how to configure robots.ts and sitemaps, how to keep HTML crawlable, and what llms.txt does — and does not — guarantee. How LLMs find your site Treat “appearing in an LLM” as three separate pipelines: Pipeline What it does Typical bots / tokens Training crawl Collects public pages that may enter future model training GPTBot, ClaudeBot, Google-Extended (token), Common Crawl’s CCBot Search / answer index Builds or refreshes retrieval so answers can cite your URLs OAI-SearchBot, Claude-SearchBot, PerplexityBot, classic Googlebot / Bingbot (and partners) User-triggered fetch Downloads a specific URL because a human asked for it (or pasted a link) ChatGPT-User, Claude-User, Perplexity-User Blocking the training bot does not automatically block the search bot. OpenAI states this explicitly for GPTBot vs OAI-SearchBot. Anthropic documents the same split for ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity documents PerplexityBot for search indexing and Perplexity-User for live fetches. ChatGPT Search can also partner with third-party search providers. OpenAI’s help center documents that rewritten queries may be sent to partners such as Bing (and others listed in that article). Independently, OpenAI recommends allowing OAI-SearchBot if you want to appear in ChatGPT search answers. Practical implication: keep search bots allowed and stay healthy in major web indexes — do not optimize for a single rumor about which partner is “really” used this month. OpenAI: the bots that decide ChatGPT visibility From OpenAI’s crawler overview: OAI-SearchBot — citations in ChatGPT Search Used to surface websites in ChatGPT’s search features. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as plain navigational links. OpenAI recommends allowing it in robots.txt and permitting its published IP ranges. Changes can take about 24 hours to propagate. GPTBot — training, not Search opt-out Crawls content that may be used to train generative foundation models. Disallowing GPTBot signals that content should not be used for that training. It does not control ChatGPT Search eligibility. ChatGPT-User — live, user-initiated fetches Used when ChatGPT or Custom GPTs fetch a page because of a user action. OpenAI notes it is not used for automatic web crawling, not used to decide Search inclusion, and that robots.txt rules may not apply because the fetch is user-initiated. Manage Search with OAI-SearchBot; treat ChatGPT-User as a separate live-fetch channel. OAI-AdsBot Only visits pages submitted as ads on ChatGPT; not used to train foundation models. Relevant if you run ChatGPT ads — ignore it for organic GEO. A common, defensible policy for content sites that want citations but not training: User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / Other answer engines (same idea, different names) Anthropic (Claude) Anthropic’s help center (updated April 2026) defines three bots: ClaudeBot — possible contribution to model training Claude-SearchBot — indexes content for search quality; disabling it may reduce visibility in search results Claude-User — user-directed fetches; disabling it may reduce visibility for user-directed web search Anthropic honors robots.txt (including non-standard Crawl-delay) and warns that IP-blocking alone is unreliable because it can prevent the bot from reading your robots.txt. Perplexity Perplexity’s crawler docs recommend allowing PerplexityBot so your site can appear in Perplexity search results. Perplexity-User handles user-initiated fetches and generally ignores robots.txt because a person requested the page. If you use a WAF, whitelist by user-agent + published IP JSON (perplexitybot.json / perplexity-user.json) — robots.txt alone is not enough when the edge drops the request. Google Gemini vs Google Search Google-Extended is a robots.txt control token, not a separate HTTP user-agent. It governs whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps / Vertex AI Grounding with Google Search. Google states it does not affect inclusion or ranking in Google Search. Important trade-off: for OpenAI you can allow search and disallow training separately. For Google, Google-Extended covers both Gemini training and Gemini grounding. Allow it if you want Gemini apps to ground on your content; disallow it if you want to opt out of those
You blocked GPTBot in robots.txt to keep your content out of training runs — and then wondered why ChatGPT Search never cites your docs. Those are different systems. OpenAI’s own crawler docs say each bot is independent: allowing OAI-SearchBot keeps you eligible for ChatGPT search answers while disallowing GPTBot opts you out of foundation-model training. This article is a Next.js App Router playbook for Generative Engine Optimization (GEO): how answer engines discover pages, which user-agents actually matter, how to configure robots.ts and sitemaps, how to keep HTML crawlable, and what llms.txt does — and does not — guarantee. How LLMs find your site Treat “appearing in an LLM” as three separate pipelines: Pipeline What it does Typical bots / tokens Training crawl Collects public pages that may enter future model training GPTBot, ClaudeBot, Google-Extended (token), Common Crawl’s CCBot Search / answer index Builds or refreshes retrieval so answers can cite your URLs OAI-SearchBot, Claude-SearchBot, PerplexityBot, classic Googlebot / Bingbot (and partners) User-triggered fetch Downloads a specific URL because a human asked for it (or pasted a link) ChatGPT-User, Claude-User, Perplexity-User Blocking the training bot does not automatically block the search bot. OpenAI states this explicitly for GPTBot vs OAI-SearchBot. Anthropic documents the same split for ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity documents PerplexityBot for search indexing and Perplexity-User for live fetches. ChatGPT Search can also partner with third-party search providers. OpenAI’s help center documents that rewritten queries may be sent to partners such as Bing (and others listed in that article). Independently, OpenAI recommends allowing OAI-SearchBot if you want to appear in ChatGPT search answers. Practical implication: keep search bots allowed and stay healthy in major web indexes — do not optimize for a single rumor about which partner is “really” used this month. OpenAI: the bots that decide ChatGPT visibility From OpenAI’s crawler overview: OAI-SearchBot — citations in ChatGPT Search Used to surface websites in ChatGPT’s search features. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as plain navigational links. OpenAI recommends allowing it in robots.txt and permitting its published IP ranges. Changes can take about 24 hours to propagate. GPTBot — training, not Search opt-out Crawls content that may be used to train generative foundation models. Disallowing GPTBot signals that content should not be used for that training. It does not control ChatGPT Search eligibility. ChatGPT-User — live, user-initiated fetches Used when ChatGPT or Custom GPTs fetch a page because of a user action. OpenAI notes it is not used for automatic web crawling, not used to decide Search inclusion, and that robots.txt rules may not apply because the fetch is user-initiated. Manage Search with OAI-SearchBot; treat ChatGPT-User as a separate live-fetch channel. OAI-AdsBot Only visits pages submitted as ads on ChatGPT; not used to train foundation models. Relevant if you run ChatGPT ads — ignore it for organic GEO. A common, defensible policy for content sites that want citations but not training: User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / Other answer engines (same idea, different names) Anthropic (Claude) Anthropic’s help center (updated April 2026) defines three bots: ClaudeBot — possible contribution to model training Claude-SearchBot — indexes content for search quality; disabling it may reduce visibility in search results Claude-User — user-directed fetches; disabling it may reduce visibility for user-directed web search Anthropic honors robots.txt (including non-standard Crawl-delay) and warns that IP-blocking alone is unreliable because it can prevent the bot from reading your robots.txt. Perplexity Perplexity’s crawler docs recommend allowing PerplexityBot so your site can appear in Perplexity search results. Perplexity-User handles user-initiated fetches and generally ignores robots.txt because a person requested the page. If you use a WAF, whitelist by user-agent + published IP JSON (perplexitybot.json / perplexity-user.json) — robots.txt alone is not enough when the edge drops the request. Google Gemini vs Google Search Google-Extended is a robots.txt control token, not a separate HTTP user-agent. It governs whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps / Vertex AI Grounding with Google Search. Google states it does not affect inclusion or ranking in Google Search. Important trade-off: for OpenAI you can allow search and disallow training separately. For Google, Google-Extended covers both Gemini training and Gemini grounding. Allow it if you want Gemini apps to ground on your content; disallow it if you want to opt out of those