Output
robots.txtHow to use this tool
Build a basic robots.txt file that allows crawling except for the paths you specify. Add your sitemap URL so supporting crawlers can find it. Review existing rules before replacing a file: robots.txt controls cooperative crawling, not access to private information.
- Enter the absolute sitemap URL and paths crawlers should skip. Leave paths empty to allow all paths.
- Merge these rules with any intentional, crawler-specific rules already on your site.
- Serve the file as plain text at your domain’s /robots.txt and check important URLs after publishing.
Example output
Illustrative Example app data. Replace these values with your own verified details.
User-agent: * Disallow: /api/ Disallow: /internal/ Sitemap: https://example.com/sitemap.xml
Common questions
Does Disallow remove a page from search results?
Not necessarily. A blocked URL can still be known from links elsewhere. Use authentication for private content. For public pages you want excluded from indexing, allow crawling so the crawler can read a noindex directive.
Does allowing AI search also allow model training?
Some providers offer separate crawlers. OpenAI documents OAI-SearchBot for search and GPTBot for training. This basic wildcard file does not set a separate training policy; preserve or add provider-specific rules according to your preferences.
Guide: Robots.txt Examples: Disallow All, AI Crawlers and Noindex
Reference: Google: robots.txt introduction
Check your published changes
After publishing, use the search and AI visibility checker to inspect your public HTML and crawler access. It reports technical signals; it cannot predict indexing, rankings, or AI recommendations.
Building a useful app? See what a reviewed UsefulShelf listing includes.