Submit appSign in

Free SEO tool · No sign-in

Robots.txt generator

Build a basic robots.txt file that allows crawling except for the paths you specify. Add your sitemap URL so supporting crawlers can find it. Review existing rules before replacing a file: robots.txt controls cooperative crawling, not access to private information.

Your inputs are processed in this browser. This tool does not fetch the URLs you enter.

Your page details

How to use the robots.txt generator

  1. Enter the absolute sitemap URL and paths crawlers should skip. Leave paths empty to allow all paths.
  2. Merge these rules with any intentional, crawler-specific rules already on your site.
  3. Serve the file as plain text at your domain’s /robots.txt and check important URLs after publishing.

Example output

Illustrative Example app data. Replace these values with your own verified details.

User-agent: *
Disallow: /api/
Disallow: /internal/

Sitemap: https://example.com/sitemap.xml

Common questions

Does Disallow remove a page from search results?

Not necessarily. A blocked URL can still be known from links elsewhere. Use authentication for private content. For public pages you want excluded from indexing, allow crawling so the crawler can read a noindex directive.

Does allowing AI search also allow model training?

Some providers offer separate crawlers. OpenAI documents OAI-SearchBot for search and GPTBot for training. This basic wildcard file does not set a separate training policy; preserve or add provider-specific rules according to your preferences.

Reference: Google: robots.txt introduction.

Check your published changes

After publishing, use the search and AI visibility checker to inspect your public HTML and crawler access. It reports technical signals; it cannot predict indexing, rankings, or AI recommendations.

Building a useful app? See what a reviewed UsefulShelf listing includes.