GeoEngine
All articles

How GPTBot and PerplexityBot read your website: from crawl request to usable source

An AI crawler does not automatically see your website as a visitor does. It requests a URL, receives a server response and processes the content that is technically available. An attractive interface offers little help when your most important information only appears after a click or through JavaScript that is not executed.

For business owners, SMEs and marketing agencies, this distinction is particularly important: being crawled, being used for model training and being cited in an AI answer are different processes. Each process has different requirements.

GPTBot and PerplexityBot do not have the same role

“AI bot” is an umbrella term. Not every bot collects information for the same purpose.

  • GPTBot is OpenAI’s crawler, which can collect content to improve and train models. Allowing GPTBot access does not automatically mean visibility in ChatGPT answers.
  • OAI-SearchBot focuses on OpenAI’s search functionality. For discoverability through ChatGPT search results, this is a different relevant bot from GPTBot.
  • PerplexityBot crawls web content for Perplexity’s search functionality. Allowing access can help make content available as a search source, but does not guarantee a mention.
  • User-driven agents, such as ChatGPT-User and Perplexity-User, can retrieve pages in response to a user query. This is different from automatically crawling the web.

Check providers’ current documentation before changing access rules. Names, functions and technical instructions can change.

Step 1: the crawler must find your url

A page can only be retrieved when its URL is known. Crawlers and search systems can discover URLs through internal links, external references, sitemaps and existing search indexes. The exact routes a service uses vary.

Avoid making discovery dependent on a single channel:

  • Link to important pages from your navigation or relevant overview pages.
  • Use standard HTML links with a valid href attribute.
  • Keep your XML sitemap up to date.
  • Ensure valuable pages are not accessible only through an internal search function.

A sitemap is a discovery aid, not an instruction to crawl or index every URL. An IndexNow notification to participating search engines is likewise not a direct instruction to all AI bots.

Step 2: access is checked at a technical level

Your server, robots.txt, CDN and security settings then come into play. A bot may be welcome according to robots.txt yet still be blocked by a firewall or CAPTCHA.

If you want to block GPTBot but allow PerplexityBot access, a robots.txt snippet could look like this:

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Allow: /

This example only governs these two named crawlers. It says nothing about OAI-SearchBot or other agents. Always check the entire file for conflicting or unintended rules.

Robots.txt is a crawling instruction, not an access security measure. Confidential information belongs behind authentication. Blocking access also does not automatically remove previously collected content.

Check HTTP status codes as well. A 403 denies access, a 429 indicates too many requests and a 5xx indicates a server problem. These responses can prevent a crawler from receiving your content.

Step 3: the crawler processes what is available

The crucial question is not just: “Is the text on my website?” The question is: “What text does this crawler actually receive?”

With server-side rendering, the main content is already present in the initial HTML response. With client-side rendering, that content may only appear after JavaScript has been executed. Support for this varies by crawler and retrieval route.

Do not assume full browser functionality. Wherever possible, make essential information directly available in HTML:

  • What your business does and who it serves.
  • Services, products and relevant terms.
  • Company name and contact details.
  • Specific answers to frequently asked questions.
  • Links to more detailed information.

An accordion is not necessarily a problem when the text is already in the HTML. Content that is only retrieved from the server after a click is less reliably accessible.

Step 4: the content must be understandable on its own

After retrieval, a system must be able to recognise what a page is about. Clear headings, descriptive page titles and specific wording help with this.

Compare these two sentences:

  • “We offer smart solutions for ambitious businesses.”
  • “We install and maintain ventilation systems for shops and offices in Antwerp.”

The second sentence identifies the service, target audience and region. This makes it easier to interpret without additional context.

Write important passages as though they could be read independently of the rest. Name the subject explicitly, explain technical terms and place conditions alongside the claim they relate to.

This makes content more useful for both people and systems that select relevant passages. It does not compel a citation.

What role do Schema.org and llms.txt play?

Schema.org JSON-LD can explicitly identify which organisation, service or other subject a page describes. Use accurate data that matches the visible content. Structured data does not replace substantive explanations.

The way you add schema deserves attention. JSON-LD injected through JavaScript is only available to systems that execute that injection or receive the rendered output.

GeoEngine is a Belgian B2B SaaS GEO and AI toolkit for SMEs and marketing agencies. The lightweight GeoEngine tag dynamically injects Schema.org JSON-LD on each page without modifying the source code. With dynamic implementations, therefore, also check what the intended crawler can process.

An llms.txt file can also provide a reading guide. It is not a universally supported access rule and does not replace robots.txt or an XML sitemap.

Check the entire chain, not just your content

A practical technical check consists of five steps:

  1. Review the server response: does the page return the correct status code?
  2. Inspect the original HTML: is the main content already present?
  3. Check access rules: do robots.txt, the CDN or the firewall block relevant bots?
  4. Analyse server logs: which URLs are requested, and with what results?
  5. Validate structured data: does the markup match the visible information?

A user-agent name alone does not prove that a request genuinely comes from a provider. Where available, use that provider’s official verification instructions.

GeoEngine offers sitemap sync, IndexNow pings to search engines, daily technical monitoring and Auto-Healing of schema. These features support technical maintenance. They do not guarantee rankings or AI citations.

Frequently asked questions

Does allowing GPTBot access ensure that ChatGPT mentions my business?

No. GPTBot access relates to a different function from search visibility. A mention depends on factors including the query, available sources and source selection.

Should i allow all AI bots?

No. Define separate policies for training, search functionality and user-driven requests. Then check each provider’s instructions.

Can a crawler read my javascript content?

It varies. Wherever possible, make important information available in the original HTML and test the specific retrieval route.

Does a correct technical setup guarantee an AI citation?

No. Technical accessibility helps systems reach your content, but does not guarantee its selection as a source.

Would you like to start with a baseline assessment? At geoengine.tech, you can request a free GEO scan as a starting point for targeted improvements.