AI Search & GEO

What Is ChatGPT-User Crawler?

By BacklinkPrice Team · Nov 18, 2026 · Updated Nov 20, 2026 · 10 min read
Illustrated cover for What Is ChatGPT-User Crawler?

ChatGPT-User is the user agent OpenAI sends when a person asks ChatGPT to open a web page. It fetches one page in real time, and robots.txt rules may not apply.

ChatGPT-User is the user agent OpenAI sends when a person using ChatGPT asks it to open a web page. It fetches one page in real time because a user asked a question that needed it, and OpenAI says robots.txt rules may not apply to that visit.

Key takeaways

  • ChatGPT-User fetches a page only when a person asks ChatGPT or a Custom GPT a question that needs it (OpenAI, 2026).
  • OpenAI says robots.txt rules may not apply to ChatGPT-User, because the visit starts with a user (OpenAI, 2026).
  • ChatGPT-User does not decide whether your pages appear in ChatGPT search. OAI-SearchBot does that job (OpenAI, 2026).
  • Its user agent string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot (OpenAI, 2026).
  • OpenAI publishes the IP ranges ChatGPT-User uses at openai.com/chatgpt-user.json, so you can check a log hit (OpenAI, 2026).
  • Cloudflare changed its default on July 1, 2025, to block AI crawlers unless they pay creators (Cloudflare, 2025).

What ChatGPT-User is

ChatGPT-User is a user agent OpenAI sends for certain user actions in ChatGPT and Custom GPTs. When a user asks ChatGPT or a Custom GPT a question, it may visit a web page with this agent, and the visit happens because the user asked.

A typical trigger is a person pasting your URL into ChatGPT and asking for a summary, or asking a question that only your page can answer. Custom GPTs can trigger the same kind of visit, and so can GPT Actions, which let ChatGPT users interact with outside apps.

ChatGPT-User is not a search crawler and not a training crawler. It does not build an index, and it does not collect pages to train a model.

That work belongs to two other OpenAI agents, OAI-SearchBot and GPTBot, and whether AI crawlers follow links on the pages they fetch is a separate question.

ChatGPT search launched on October 31, 2024, after the SearchGPT prototype of July 25, 2024. It opened to all logged-in users on December 16, 2024, and to signed-out users on February 5, 2025.

How ChatGPT-User differs from OpenAI's other agents

OpenAI documents four agents as of November 2026, and each one has its own job, user agent string and IP list. The table below shows what each one does, and the rest of this section explains the two distinctions that matter most.

AgentJobRuns on its own?robots.txt
OAI-SearchBotSurfaces sites in ChatGPT search resultsYesFollowed; blocking it removes you from search answers
GPTBotCrawls content that may train foundation modelsYesFollowed; blocking it signals no training use
ChatGPT-UserFetches a page when a user asks ChatGPT or a Custom GPTNo, user-triggeredOpenAI says rules may not apply
OAI-AdsBotChecks landing pages submitted as ChatGPT adsNo, visits submitted pages onlyOnly visits pages submitted as ads

Each setting is independent of the others. OpenAI gives the example of allowing OAI-SearchBot so your pages can appear in search results while disallowing GPTBot so your content is not used for training, and if you allow both, OpenAI may reuse one crawl for both uses.

The OAI-SearchBot setting is the one that controls search visibility. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

After you change robots.txt, allow about 24 hours before you judge the result. OpenAI says it can take around 24 hours from a robots.txt update for its systems to adjust for search results, so a test run five minutes after saving proves nothing.

Why ChatGPT-User matters for your site

ChatGPT-User matters because it is the agent that puts your page in front of a person at the exact moment they ask about it.

A pasted link, a pricing question or a request to check a claim can all send it to your site, and the page it reads shapes the answer ChatGPT gives.

That visit is not the same as a citation. ChatGPT search draws on an index built by OAI-SearchBot, so a page fetched for one user may never appear as a cited source. Read how ChatGPT chooses which sources to cite to see what turns a retrieved page into one.

Ahrefs studied 1.4 million ChatGPT prompts in February 2026 and found that ChatGPT cited 88% of the pages it pulled from ordinary web search, but only 12% from news and under 2% from Reddit, YouTube and academic papers.

Pages from the search index are the ones that get cited, so search visibility is the gate.

Titles matter. In the same Ahrefs study, cited pages had titles closer in meaning to the user question, with a similarity score of 0.60 against 0.48 for pages that were retrieved but not cited.

In an Ahrefs study of 17 million citations, ChatGPT cited URLs that were 458 days newer than the results Google returned for the same queries.

ChatGPT also rewrites questions before it searches. In Ahrefs' study of 66.6 million responses from free, logged-out ChatGPT in August and September 2026, responses with fan-out queries had about 1.6 each, so one question can pull pages for several smaller questions at once.

Ranking for the topic helps even when the exact page does not rank. In an Ahrefs study from 2025, 31.8% of ChatGPT citations came from a website that ranks on Google for the question, but only 10% were the exact page that ranks.

A site that covers a topic well can be cited through a page that never ranked.

The audience is large and still growing. In November 2026, OpenAI's consumer product lead told Big Technology that ChatGPT web searches add up to roughly one-fifth of Google's query volume.

OpenAI's own usage data shows about one in five consumer messages in September 2026 were people looking for information.

A visit from ChatGPT-User is one step in a longer chain. Learn what an AI citation is to see how a fetched page becomes a named source in an answer.

How to check and control ChatGPT-User visits

You can check ChatGPT-User in under an hour with tools you already have. The work is reading logs, confirming the sender, and then writing robots.txt rules for the bots you can control.

Step 1: Find ChatGPT-User in your server logs

Open your raw server logs and search for the token ChatGPT-User. The full user agent string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot, and matching on the token ChatGPT-User is enough.

Step 2: Confirm the visit is really from OpenAI

Anyone can copy a user agent string, so check the source IP address. OpenAI publishes the IP ranges ChatGPT-User uses at openai.com/chatgpt-user.json, and a hit from outside those ranges is not OpenAI, no matter what the user agent says.

Step 3: Write robots.txt rules for the bots you can control

You cannot reliably stop ChatGPT-User with robots.txt, because OpenAI says the rules may not apply to user-initiated visits. What you can control is GPTBot and OAI-SearchBot, which both follow robots.txt, so write one rule per bot and test it after about 24 hours.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This example blocks training use while keeping your pages in ChatGPT search. If you want out of search answers too, disallow OAI-SearchBot as well, and expect to keep appearing only as a navigational link.

Step 4: Check your CDN and firewall before you blame robots.txt

A firewall or CDN can block a bot before your server ever sees it, so robots.txt is not the only gate.

Cloudflare changed its default on July 1, 2025, to block AI crawlers unless they pay creators, so check your Cloudflare AI bot settings if OpenAI agents stop arriving.

Tools, time and cost

You need no paid tool to check ChatGPT-User. Raw server logs, your hosting control panel and a text search are enough for the check, and the whole job takes about 30 to 60 minutes on a small site.

ToolWhat it tells youTime needed
Raw server logsEvery ChatGPT-User hit, with IP, page and timeAbout 30 minutes
OpenAI IP listWhether a hit really came from OpenAIAbout 5 minutes
Cloudflare dashboardWhether AI bots are blocked at the edgeAbout 10 minutes
Ahrefs Brand RadarWhich prompts mention and cite your siteOngoing

Paid AI visibility tools can track citations across prompts, but check their pricing pages before you buy, since prices change. For the basic check in this guide, logs and the free IP list are all you need.

Common mistakes

  • Blocking GPTBot and expecting to vanish from ChatGPT answers. GPTBot collects training data, and blocking it does not remove you from search answers. OAI-SearchBot controls search visibility, and the two settings are independent.
  • Writing a ChatGPT-User rule in robots.txt and trusting it. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user started the visit. A Disallow for ChatGPT-User is a request, not a lock.
  • Blocking OAI-SearchBot while asking why citations stopped. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Allow it if you want to be cited, and allow about 24 hours after a change.
  • Forgetting the CDN layer. A robots.txt allow means nothing if your firewall blocks the bot first, so check your CDN settings too. Cloudflare changed its default on July 1, 2025, to block AI crawlers unless they pay creators.
  • Treating one ChatGPT-User hit as a citation. A fetch means one person asked about your page, not that ChatGPT cited it in search results. Citations come from the search index, so a hit in your logs is a lead, not proof of visibility.

How to measure results

Measure ChatGPT-User in your logs, not in your rankings. Count ChatGPT-User hits per week, note which pages it fetches, and watch whether the count changes after you edit robots.txt or your CDN settings.

For citations, track mentions instead of hits. Ahrefs Brand Radar shows which prompts mention and cite your site, so compare the prompts that cite you before and after you allow OAI-SearchBot. Give each change about 24 hours, since OpenAI says its systems take that long to adjust.

Put the numbers in context with what AI visibility is, which explains how mentions, citations and traffic fit together.

Cloudflare's own numbers show why the stakes are high. In its July 2025 post, Cloudflare said getting traffic from OpenAI is 750 times more difficult than it was with the Google of old, and 30,000 times more difficult with Anthropic.

A page that is fetched but never cited earns you nothing back.

Also watch your training opt-out separately from search. GPTBot is the most-blocked AI crawler in Ahrefs' study of about 140 million websites, so many sites already block training while staying in search.

Check that your GPTBot rule and your OAI-SearchBot rule say what you mean, since the two settings are independent.

Frequently Asked Questions

Does ChatGPT-User respect robots.txt?

Not reliably. OpenAI says ChatGPT-User is not used for crawling the web in an automatic fashion, and because a user starts the visit, robots.txt rules may not apply. If you want to control what ChatGPT can use, set rules for GPTBot and OAI-SearchBot instead, since both of those follow robots.txt.

Will blocking ChatGPT-User remove my site from ChatGPT answers?

No. ChatGPT-User does not decide whether your content appears in search, and OpenAI says to use OAI-SearchBot in robots.txt for search opt-outs. Blocking ChatGPT-User may stop some live fetches, but it will not remove your pages from ChatGPT search answers.

What is the difference between ChatGPT-User and OAI-SearchBot?

OAI-SearchBot crawls the web on its own to build the index behind ChatGPT search, and blocking it removes your pages from search answers. ChatGPT-User only fetches a page when a person asks ChatGPT or a Custom GPT a question that needs it, and it does not build an index.

How do I verify a ChatGPT-User visit is really from OpenAI?

Check the source IP address against the list OpenAI publishes at openai.com/chatgpt-user.json. The user agent string alone proves nothing, since anyone can copy it, but a hit from an IP outside the published ranges is not OpenAI. Check both the token and the IP before you act on a log entry.

Does ChatGPT-User send traffic to my site?

It can, when a person clicks through from a ChatGPT answer to read your page, but many fetches end in the chat window. The only number you can trust is your own: count ChatGPT-User hits in your logs and compare them with the visits your analytics records from ChatGPT.

Can I allow ChatGPT-User but block GPTBot?

Yes. OpenAI says each setting is independent, so you can disallow GPTBot to keep your content out of training while leaving OAI-SearchBot allowed for search. ChatGPT-User visits are user-triggered, so a rule aimed at it may not change what happens when a person asks ChatGPT to open your page.

Sources

If you want more of your pages cited in ChatGPT answers, see our pricing for link building that earns mentions on real sites.

BacklinkPrice Team
Written by
BacklinkPrice Team
Link building and backlink pricing research

The BacklinkPrice Team researches backlink prices, publisher sites and link building tools. We check prices and plans against each provider's own pages and date them, so you know how current the numbers are.

More from BacklinkPrice Team

Related guides