AI Search & GEO

Do AI Crawlers Follow Links?

By BacklinkPrice Team · Nov 18, 2026 · Updated Nov 20, 2026 · 11 min read
Illustrated cover for Do AI Crawlers Follow Links?

Yes, AI crawlers follow links, but each crawler does it differently. Google documents link-based discovery, while OpenAI and Perplexity split their bots by job.

Yes, AI crawlers follow links, but not every crawler does it the same way, and a link never guarantees that the target page will be fetched or cited. Google documents link-based discovery in detail, while OpenAI and Perplexity describe search crawlers without stating how they find each link.

Key takeaways

  • Googlebot discovers new pages by extracting links from pages it already knows, and it also uses sitemaps (Google Search Central, 2026).
  • OpenAI runs 4 separate agents, and only OAI-SearchBot decides whether a page can appear in ChatGPT search answers (OpenAI, 2026).
  • PerplexityBot surfaces and links websites in Perplexity results, and Perplexity says it is not used to crawl content for AI foundation models (Perplexity, 2026).
  • None of the major AI crawlers rendered JavaScript in Vercel and MERJ tests from December 2024, so links that only appear after JavaScript runs stay hidden from them.
  • A robots.txt change takes about 24 hours to affect OpenAI search results and up to 24 hours for Perplexity, as of November 2026.

The short answer

Yes, with one split to keep in mind. Search crawlers follow links to find pages, while training crawlers collect pages they already know about, and user-triggered fetchers open one page because a person asked about it.

Google states the link part plainly. Its guide says some pages are discovered when Google extracts a link from a known page to a new page, and it gives a category page linking to a new blog post as the example (Google Search Central).

OpenAI and Perplexity do not state the same detail. Their crawler pages say what each bot is for and how robots.txt controls it, but neither page says whether the bot extracts links from the pages it fetches. Treat link-following by AI-only search crawlers as likely but not documented, and build your site so Googlebot can follow the same links.

An AI crawler is an automated program that requests web pages for an AI company. Following a link means reading a link on a fetched page, adding the target URL to a list of pages to fetch, and then requesting it.

Three different jobs get mixed up under one name. Discovery is learning that a URL exists. Fetching is requesting the page. Citing is naming the page in an answer, which needs the page to be fetched and held in a search index first.

  • Discovery: a crawler learns a URL from a link, a sitemap, or a submission.
  • Fetching: a crawler requests the page and reads the HTML it gets back.
  • Citing: an answer engine names the page as a source, with a link, in its reply.

A link can help with the first two jobs and still fail at the third. A page that is found and fetched may still not be cited, because the answer engine picks sources by relevance to the question, not by how many links point at the page.

The crawlers below do not share one rulebook. Each operator publishes its own list of bots, and each bot answers to its own robots.txt group.

CrawlerJobLink behaviour as documented
GooglebotSearch index for Google SearchDiscovers new pages by extracting links from known pages
OAI-SearchBotSearch index for ChatGPT searchOpenAI does not state how it finds links
GPTBotTraining data for OpenAI modelsOpenAI does not state how it finds links
PerplexityBotSearch index for PerplexityPerplexity does not state how it finds links
ChatGPT-UserFetches a page a user asked aboutOpens a specific page, does not crawl the web
Perplexity-UserFetches a page a user asked aboutOpens a specific page, does not crawl the web

Googlebot is the clearest case. Google Search Central says there is no central registry of all web pages, so Google must constantly look for new pages, and one way it finds them is by extracting a link from a known page. Google also says its crawlers use links as a signal for relevance and to find new pages to crawl.

The link itself must be written in a form the crawler can parse. Google Search Central says it can generally only crawl a link that is an <a> element with an href attribute, and most links in other formats will not be parsed. Put the target URL in a plain href, and use descriptive anchor text between the tags.

Qualified links are the exception. Google says links marked with sponsored, ugc, or nofollow will generally not be followed, though the target page may still be found through a sitemap or another link. The three values are explained in what is a nofollow link.

OpenAI splits its crawlers by job. OpenAI lists 4 agents: OAI-SearchBot for search, GPTBot for training, ChatGPT-User for user actions, and OAI-AdsBot for ad landing pages. Each setting is independent, so a site can allow OAI-SearchBot while disallowing GPTBot, and OpenAI may use one crawl for both jobs when both are allowed.

OAI-SearchBot is the bot that decides ChatGPT search visibility. OpenAI says sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. For detail on what ChatGPT does with the pages it finds, see how ChatGPT chooses which sources to cite.

PerplexityBot works the same way at the level OpenAI describes. Perplexity says PerplexityBot is designed to surface and link websites in search results on Perplexity, and it is not used to crawl content for AI foundation models. Perplexity recommends allowing it in robots.txt and permitting its published IP ranges. Its source rules are covered in how Perplexity chooses sources.

User-triggered fetchers do not follow links at all. OpenAI says ChatGPT-User is not used for crawling the web in an automatic fashion, and Perplexity says Perplexity-User is not used for web crawling. Both open a page because a person asked a question that needs it, so a link in an answer can still lead to a fetch of that page.

Google's own AI answers sit on top of Googlebot, not on a separate AI crawler. That is why the same link rules apply to them, and how Google AI Overviews choose sources explains what happens after a page is found.

Links are the discovery path a new page depends on. Without a link, a sitemap entry, or a submission, a crawler may never learn that the page exists, and a page that is never fetched cannot be cited.

That order is the core of generative engine optimization (GEO): a page must be found, fetched, and held in an index before an answer engine can cite it.

Scale makes the point concrete. Vercel and MERJ counted 569 million GPTBot fetches, 370 million Claude fetches, 314 million AppleBot fetches, and 24.4 million PerplexityBot fetches in one month, against 4.5 billion Googlebot fetches (Vercel and MERJ, December 2024). AI crawlers are busy, but they still fetch far less than Googlebot.

JavaScript is the biggest trap. The same Vercel and MERJ study found that none of the major AI crawlers rendered JavaScript, while Googlebot renders pages with a recent version of Chrome. A link that only exists after JavaScript runs is visible to Google and hidden from most AI crawlers.

The traffic is still growing fast. Cloudflare's 2025 analysis found AI and search crawler traffic rose 18% from May 2024 to May 2025, and GPTBot requests rose 305% in the same period (Cloudflare, 2025). A page that is easy to discover now is easy to discover while that growth continues.

Make every link a plain HTML link first, then check that each bot can reach it. The steps below need a text editor, your server logs, and access to robots.txt and your sitemap.

Write links as <a> elements with an href attribute that holds a real URL. Google says most links in other formats will not be parsed, and it cannot reliably extract URLs from tags that act as links only through script events.

Put navigation, category links, and links to your main guides in the HTML the server sends, not in code that runs in the browser. Vercel and MERJ found in December 2024 that ChatGPT and Claude fetch JavaScript files without executing them, so content that needs JavaScript stays unread.

Google advises that every page you care about should have a link from at least one other page on your site. Add contextual links from related guides, and check that deep pages are not reachable only from a sitemap. If a page has no incoming link, treat that as a gap to fix.

Step 4: Keep a sitemap as a second path, not the only path

Submit a sitemap as well as linking well. Google lists sitemaps as a discovery path next to links, but a sitemap entry is a hint, not a guarantee of a crawl. The link and the sitemap entry should point at the same canonical URL.

Step 5: Allow the search bots in robots.txt

Check robots.txt for each bot by name, because one rule does not cover them all. Allow OAI-SearchBot and PerplexityBot if you want to appear in ChatGPT and Perplexity answers, and decide on GPTBot separately, since blocking it only opts you out of training.

Allow about a day for the change to take effect. OpenAI says a robots.txt change can take about 24 hours to affect search results, and Perplexity says its systems can take up to 24 hours to reflect changes, as of November 2026.

Step 6: Fix redirects and dead ends

Vercel and MERJ found in December 2024 that 34.82% of ChatGPT crawler fetches hit 404 pages and 14.36% followed redirects, against 8.22% and 1.49% for Googlebot. Check your logs for the same pattern, fix broken internal links, and keep redirects to one hop where you can. For a full check, see how to get backlinks indexed.

Tools, time and cost

You need no paid tool for the checks in this guide. A text editor opens robots.txt and your sitemap, your server logs show which bots arrived and what they asked for, and Google Search Console shows how Google sees a page.

Google's link guide points to its URL Inspection Tool to check that anchor text added by JavaScript is present in the rendered HTML. Use it after any change to a template that builds links with code.

  • Server logs: filter by user agent token, such as OAI-SearchBot or PerplexityBot, and look for 404s and redirect chains.
  • robots.txt: one group per bot, with search bots allowed and training bots decided on their own.
  • Sitemap: one file, kept current, with canonical URLs only.
  • Time: the first pass over a small site fits in a single work session, and re-checks take minutes when a template changes.

Common mistakes

  • Blocking the wrong bot: disallowing GPTBot removes training use only, while blocking OAI-SearchBot removes ChatGPT search answers, so name each bot in its own robots.txt group.
  • Hiding links in JavaScript menus: AI crawlers do not run JavaScript, so a menu built only in code hides every link behind it from them.
  • Marking internal links nofollow: Google says links marked nofollow will generally not be followed, so keep the value for paid, untrusted, or user-submitted links.
  • Expecting a link to force a citation: discovery and fetching come first, and the answer engine still picks sources by relevance to the question.
  • Trusting a sitemap alone: a sitemap is a hint, and a page with no incoming link is easy for a crawler to miss when the sitemap is stale.

Frequently Asked Questions

Google says links marked nofollow, sponsored, or ugc will generally not be followed, though the target may still be found another way. OpenAI and Perplexity publish no rule of their own for nofollow, so treat a qualified link as a link an AI crawler will usually skip. Use the value only where Google advises it.

Most do not. Vercel and MERJ found in December 2024 that none of the major AI crawlers rendered JavaScript, so a link that only appears after code runs stays hidden from them. Googlebot is the exception, since it renders pages with a recent version of Chrome. Server-render the links that matter.

A link can help ChatGPT find your page, but it cannot force a citation. OpenAI says OAI-SearchBot surfaces sites in ChatGPT search, and sites that opt out of it will not be shown in search answers. After discovery, the page must be fetched, indexed, and picked as relevant to the question.

Google discovers pages from links on any page it already knows, whether the link points inside your site or to another site. OpenAI and Perplexity do not publish a rule that splits internal and external links. Treat both as discovery paths, and keep both in plain HTML.

No operator publishes a timetable for following a new link, and discovery does not guarantee a crawl. The only timing either OpenAI or Perplexity publishes is for robots.txt changes, about 24 hours for OpenAI and up to 24 hours for Perplexity. Check server logs to see when a bot actually arrives.

Sources

If you want the pages those links point to to earn links of their own, see how to build backlinks.

BacklinkPrice Team
Written by
BacklinkPrice Team
Link building and backlink pricing research

The BacklinkPrice Team researches backlink prices, publisher sites and link building tools. We check prices and plans against each provider's own pages and date them, so you know how current the numbers are.

More from BacklinkPrice Team

Related guides