Supporting AI Crawlers with ai.txt

Modern AI tools like Microsoft Copilot, ChatGPT, and other intelligent assistants often rely on structured and accessible content from your website to provide helpful answers to users. To make it easier for AI crawlers to understand and index your public help content, you can use a simple file named ai.txt.

What is ai.txt?

ai.txt is a proposed standard similar to robots.txt, but specifically designed for AI crawlers. It tells AI tools which parts of your website they are allowed to access and index for use in their models or integrations.

While not all AI tools support ai.txt today, support is growing, and adding this file can prepare your content for future AI integrations.

For example, an ai.txt file can allow AI crawlers everywhere on your site and point them to a machine-readable list of your content:

HelpGuides Support for ai.txt

HelpGuides generates an ai.txt file for your site automatically, exactly like the example above: it allows AI crawlers to access your whole site and points them to site.json, a JSON list of your published articles. As ai.txt is an emerging standard, the file HelpGuides generates may change.

If Allow Search Engine Indexing is turned off in your project's Settings, on the General tab, the file asks all crawlers to stay away instead:

Your ai.txt file is at the root of your site. For example:

https://docs.helpguides.io/ai.txt

To help AI tools further, see Using LLMS.txt, Exposing Your APIs with openapi.json and Model Context Protocol.