# Robots.txt generator

> Generate robots.txt with crawl exclusions and a sitemap URL. Learn how crawl rules differ from noindex, authentication, and AI training permissions.

Canonical: https://usefulshelf.co/tools/robots-txt-generator

Choose which paths crawlers should skip and point them to your sitemap.

## Your details

Runs in your browser. No sign-in or URL fetching. Enable JavaScript to use the form. The instructions and sample output below are available without JavaScript.

- Sitemap URL (required)
- Disallowed paths (optional, one per line) (optional). Paths must start with /. A single / blocks the entire site.

For URL fields, https:// is optional. For example, example.com.

Action: Generate robots.txt. Output filename: `robots.txt`.

## How to use this tool

Build a basic robots.txt file that allows crawling except for the paths you specify. Add your sitemap URL so supporting crawlers can find it. Review existing rules before replacing a file: robots.txt controls cooperative crawling, not access to private information.

1. Enter the absolute sitemap URL and paths crawlers should skip. Leave paths empty to allow all paths.
2. Merge these rules with any intentional, crawler-specific rules already on your site.
3. Serve the file as plain text at your domain's /robots.txt and check important URLs after publishing.

## Example output

Illustrative Example app data. Replace these values with your own verified details.

```text
User-agent: *
Disallow: /api/
Disallow: /internal/

Sitemap: https://example.com/sitemap.xml

```

## Common questions

### Does Disallow remove a page from search results?

Not necessarily. A blocked URL can still be known from links elsewhere. Use authentication for private content. For public pages you want excluded from indexing, allow crawling so the crawler can read a noindex directive.

### Does allowing AI search also allow model training?

Some providers offer separate crawlers. OpenAI documents OAI-SearchBot for search and GPTBot for training. This basic wildcard file does not set a separate training policy; preserve or add provider-specific rules according to your preferences.

Guide: [Robots.txt Examples: Disallow All, AI Crawlers and Noindex](https://usefulshelf.co/blog/robots-txt-examples).

Reference: [Google: robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro).

## Check your published changes

After publishing, use the [search and AI visibility checker](https://usefulshelf.co/check) to inspect your public HTML and crawler access. It reports technical signals; it cannot predict indexing, rankings, or AI recommendations.

Building a useful app? [See what a reviewed UsefulShelf listing includes](https://usefulshelf.co/pricing).

## Related free tools

- [Meta tag generator](https://usefulshelf.co/tools/meta-tag-generator)
- [SERP preview tool](https://usefulshelf.co/tools/serp-preview)
- [XML sitemap generator](https://usefulshelf.co/tools/xml-sitemap-generator)
- [Software schema generator](https://usefulshelf.co/tools/software-schema-generator)
- [llms.txt generator](https://usefulshelf.co/tools/llms-txt-generator)
- [UTM builder](https://usefulshelf.co/tools/utm-builder)
- [Word and character counter](https://usefulshelf.co/tools/word-counter)
- [URL slug generator](https://usefulshelf.co/tools/slug-generator)
- [FAQ schema generator](https://usefulshelf.co/tools/faq-schema-generator)
- [Mailto link generator](https://usefulshelf.co/tools/mailto-link-generator)
- [WhatsApp link generator](https://usefulshelf.co/tools/whatsapp-link-generator)

[All free SEO tools](https://usefulshelf.co/tools)