Robots.txt Checker

Robots.txt Checker is a free online tool that tests your robots.txt file in seconds. Enter your site’s URL, click “Check,” and get your robots.txt validation results instantly — no setup or sign-up required.

Robots.txt is a plain text file that tells search engine bots which parts of your site they may crawl.

To see any site’s robots.txt file, add /robots.txt to the end of the domain in your browser’s address bar for example https://yourdomain.com/robots.txt. The file is plain text, so it opens directly in the browser. Reading it shows which bots can reach which pages, which directories are closed to crawling, and whether a sitemap is declared. The same method works on competitor sites. To see how bots actually interpret those rules instead of reading them by hand, paste the address into Robots.txt Checker.

The tool fetches the robots.txt file at the domain you provide and analyzes how bots interpret it:

  • Confirms the file exists and sits in the root directory.
  • Detects syntax errors and invalid directives.
  • Parses the User-agent, Disallow, Allow, and Sitemap lines.
  • Tests whether a bot such as Googlebot can access a given URL.
  • Flags conflicting rules, incorrect paths, and missing User-agent lines.
  • Lists declared sitemaps, which you can validate with Sitemap Checker.

File format and location

  • The file must be plain text, UTF-8 encoded, and named exactly robots.txt.
  • It must sit in the root directory, at https://yourdomain.com/robots.txt. A file in a subfolder is invalid.
  • It applies only to its own host, protocol, and port: every subdomain needs its own file, and http and https are treated separately.
  • Google may ignore anything beyond 500 KiB.

Directives and rule precedence

  • Each block starts with User-agent:; the Disallow: and Allow: lines define which paths that bot may crawl.
  • User-agent: * covers all bots. An empty Disallow: applies no restriction; Allow: re-opens a subpath inside a blocked directory.
  • * matches any sequence of characters, $ marks the end of a URL, and # lines are comments.
  • Directive and bot names are case-insensitive, but paths are case-sensitive: /Admin/ and /admin/ differ.
  • When rules conflict, the most specific rule (longest path) wins; on a tie, the least restrictive one applies, so Allow wins.
  • Google does not support the crawl-delay directive; Bing and Yandex do.

Crawling and indexing behavior

  • A robots.txt returning 4xx is ignored and the site is treated as fully crawlable. A persistent 5xx can make Google temporarily treat the whole site as disallowed.
  • Google caches the file for up to 24 hours, so changes are not reflected immediately.
  • Robots.txt does not prevent indexing: a blocked URL found via external links can still appear in search results without a snippet. To keep a page out of the index, use noindex but noindex only works if the page is crawlable, so never Disallow a page you have tagged with it.

An example robots.txt file:

User-agent: *
Disallow: /admin/
Allow: /admin/public/
Sitemap: https://yourdomain.com/sitemap.xml

A stray Disallow: / line can stop your entire site from being crawled, costing you indexing and traffic. Regular robots.txt checks let you catch that risk early, confirm the file still works after a deploy, CMS update, or migration, and review how competitors configure theirs.

  1. Enter the site URL you want to check.
  2. Click the “Check” button.
  3. Review the rules, errors, and warnings in the report.
  4. Edit the file if needed and re-test.
  5. Monitor the changes through Google Search Console.

Use Disallow for admin panels, search result pages, and temporary directories — never for blog or product pages. Add your sitemap with a Sitemap: line, and re-test after every major site update. Try Robots.txt Checker free now.