{"Status":true,"Message":"","Response":{"post":{"postuid":"d373d368-bf27-4906-ab51-b674210db525","tenantuid":"d8b744fc-2e70-4089-bb80-dd1d08f6c7b2","projectuid":"fc6490ac-7527-4f49-b06e-46f701280e85","title":"Supporting AI Crawlers with ai.txt","slug":"article/supporting-ai-crawlers-with-aitxt","html":"\u003Cp\u003EModern AI tools like Microsoft Copilot, ChatGPT, and other intelligent assistants often rely on structured and accessible content from your website to provide helpful answers to users. To make it easier for AI crawlers to understand and index your public help content, you can use a simple file named \u003Ccode\u003Eai.txt\u003C/code\u003E.\u003C/p\u003E\u003Ch2 id=\u0022what_is_aitxt\u0022\u003EWhat is ai.txt?\u003C/h2\u003E\u003Cp\u003E\u003Ccode\u003Eai.txt\u003C/code\u003E is a proposed standard similar to \u003Ccode\u003Erobots.txt\u003C/code\u003E, but specifically designed for AI crawlers. It tells AI tools which parts of your website they are allowed to access and index for use in their models or integrations.\u003C/p\u003E\u003Cp\u003EWhile not all AI tools support \u003Ccode\u003Eai.txt\u003C/code\u003E today, support is growing, and adding this file can prepare your content for future AI integrations.\u003C/p\u003E\u003Cp\u003EFor example, an \u003Ccode\u003Eai.txt\u003C/code\u003E file can allow AI crawlers everywhere on your site and point them to a machine-readable list of your content:\u003C/p\u003E\u003Cdiv class=\u0022code_wrapper\u0022\u003E\u003Cdiv class=\u0022code\u0022 data-language=\u0022text\u0022\u003EUser-agent: *\nAllow: /\nData-Format:JSON\nSitemap: https://YOUR-DOMAIN/site.json\u003C/div\u003E\u003C/div\u003E\u003Ch2 id=\u0022helpguides_support_for_aitxt\u0022\u003EHelpGuides Support for ai.txt\u003C/h2\u003E\u003Cp\u003EHelpGuides generates an \u003Ccode\u003Eai.txt\u003C/code\u003E file for your site automatically, exactly like the example above: it allows AI crawlers to access your whole site and points them to \u003Ccode\u003Esite.json\u003C/code\u003E, a JSON list of your published articles. As \u003Ccode\u003Eai.txt\u003C/code\u003E is an emerging standard, the file HelpGuides generates may change.\u003C/p\u003E\u003Cp\u003EIf \u003Cb\u003EAllow Search Engine Indexing\u003C/b\u003E is turned off in your project\u0027s \u003Cb\u003ESettings\u003C/b\u003E, on the \u003Cb\u003EGeneral\u003C/b\u003E tab, the file asks all crawlers to stay away instead:\u003C/p\u003E\u003Cdiv class=\u0022code_wrapper\u0022\u003E\u003Cdiv class=\u0022code\u0022 data-language=\u0022text\u0022\u003EUser-agent: *\nDisallow: /\u003C/div\u003E\u003C/div\u003E\u003Cp\u003EYour \u003Ccode\u003Eai.txt\u003C/code\u003E file is at the root of your site. For example:\u003C/p\u003E\u003Cp\u003E\u003Ca href=\u0022https://docs.helpguides.io/ai.txt\u0022\u003Ehttps://docs.helpguides.io/ai.txt\u003C/a\u003E\u003C/p\u003E\u003Cp\u003ETo help AI tools further, see \u003Ca href=\u0022/article/using-llmstxt\u0022\u003EUsing LLMS.txt\u003C/a\u003E, \u003Ca href=\u0022/article/exposing-your-help-api-with-openapijson\u0022\u003EExposing Your APIs with openapi.json\u003C/a\u003E and \u003Ca href=\u0022/article/model-context-protocol\u0022\u003EModel Context Protocol\u003C/a\u003E.\u003C/p\u003E","publish_status":0,"post_type":"Article","authoruid":"3dde8c16-763a-4a2b-ae0b-1d8c50c62e3d","author":{"authoruid":"3dde8c16-763a-4a2b-ae0b-1d8c50c62e3d"},"featured_image_updating":false,"meta_description":"Learn how the ai.txt file helps AI tools like Copilot and ChatGPT understand your website\u0027s public content for better indexing and future AI integration.","display_toc":true,"has_workingcopy":false,"allow_indexing":true,"total_views":1001,"date_published":"2025-06-04T15:57:00","date_updated":"2026-10-04T21:50:15.563","date_created":"2025-06-04T15:48:26.35"}}}