Skip to content
WebScore LogoWebScore
AllMetadata GeneratorSchema Markup Generatorrobots.txt ToolFavicon GeneratorFavicon CheckerSSL Certificate CheckerEmail Security CheckerEmail DNS GeneratorUTM Campaign BuilderQR Code Generator

robots.txt Tool

Generate a robots.txt file, or test whether a URL is crawlable.

Configure
robots.txt
text
User-agent: *
Disallow:

Place this file at the root of your domain: https://yourdomain.com/robots.txt

Is your whole site crawlable?

Get the full WebScore report

A bad robots.txt or noindex tag can hide your pages from Google. WebScore audits indexability, sitemaps, canonical tags and more — free to run, full reports for $3.

Full detailed report with all issues and step-by-step fixes
Unlock any report for just $3 — no subscription needed
Automated daily/weekly monitoring with alerts
AI-powered UI/UX audit with annotated screenshots
Domain authority metrics from MOZ, Majestic & Ahrefs
Lifetime access for $29.99 — pay once, everything unlocked forever

Scan free, no credit card required. Unlock full reports for $3 each or $29.99 once for lifetime access.

What is robots.txt?

robots.txt is a plain-text file at the root of your domain that tells search engine crawlers which parts of your site they may or may not request. It's the first file Googlebot looks for. Used well, it keeps crawlers out of admin areas and duplicate pages and points them to your sitemap; used badly, it can accidentally hide your entire site from search.

User-agent

Which crawler the rules apply to. * means all crawlers; you can also target Googlebot, Bingbot, GPTBot, etc.

Disallow / Allow

Paths to block or explicitly permit. The most specific (longest) matching rule wins.

Sitemap

An absolute URL to your XML sitemap so crawlers can discover all your pages quickly.

Crawl-delay

Asks crawlers to wait N seconds between requests. Honored by some bots (not Google).

Common mistakes to avoid

  • Disallow: / by accident — this single line blocks your entire site. The tester above catches it instantly.
  • Using robots.txt to hide pages — blocked pages can still appear in search without a snippet. Use a noindex tag to truly de-index.
  • Forgetting the sitemap — always declare your sitemap so crawlers find new pages faster.

Audit your full crawlability

robots.txt is one piece of technical SEO. Scan your site with WebScore to check indexability, sitemaps, canonical tags, structured data and more — free to run.