SEO & webmaster
Robots.txt Tester
Find out instantly whether a page is blocked or allowed for Google.
Contents of robots.txt
Your robots.txt file tells search engines what they may crawl on your site. One badly written line can block an entire folder — or the whole site — while everything still looks perfectly normal to human visitors. This tester fetches the robots.txt of any site, shows you its full contents, then simulates a crawler: enter a path and a user-agent (say, Googlebot) and get the verdict — allowed or blocked. The simulation applies the real precedence rules, including longest-match, the * wildcard and the $ anchor, so the answer matches what Google would actually do. Free, no sign-up, and worth running before and after every deployment.
How does it work?
- Enter the site address — its robots.txt file is fetched and displayed.
- Type the path you want to test (e.g. /blog/article or /admin/).
- Pick the crawler to simulate (Googlebot, Bingbot or another).
- Click “Test” and read the verdict: allowed or blocked, with the matching rule.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
Not necessarily. Blocking prevents crawling, not indexing: if other sites link to the page, Google may still list it without a description. To keep a page out of results, use a noindex tag — and let crawlers access the page so they can read it.
What does “Disallow: /” mean?
Total blocking: the targeted crawlers may not explore the site at all. It is often a leftover from a development phase — and one of the most expensive SEO mistakes, because the site slowly vanishes from search results.
How do the * wildcard and the $ symbol work?
The asterisk matches any sequence of characters: “Disallow: /*.pdf” blocks every PDF. The dollar sign anchors the end of the URL: “Disallow: /*.pdf$” only blocks addresses ending exactly in .pdf. When rules conflict, the longest match wins, and Allow beats Disallow at equal length.
My site has no robots.txt — is that a problem?
No: without the file, search engines crawl everything, which suits most small sites. The file becomes useful to keep crawlers out of low-value areas (internal search, filters, cart) and to point them to your sitemap.