That One CrawlerDaddy mode

Indexing check

Blocked by robots.txt: what it means and how to fix it

“Blocked by robots.txt” means a Disallow rule in your site's robots.txt file stops crawlers from fetching that URL. Search engines can't read what's on the page, so it ranks poorly or not at all. Remove or narrow the rule if the page should be searchable.

By Updated · 1 min read

Why it matters

robots.txt rules are powerful and easy to over-apply: one line like “Disallow: /blog” hides a whole section. Rules copied from another site or left over from development are common culprits. Blocking also stops search engines from seeing a noindex tag on the page.

How to fix it

  1. Open yoursite.com/robots.txt and read the rules under “User-agent: *”.

  2. Find the Disallow line matching the blocked URL (the longest matching rule wins).

  3. Delete it, or narrow it so it only covers what should stay private.

  4. Keep admin, cart and internal search pages blocked if you like: they don't need to rank.

  5. Rescan to confirm important pages are no longer blocked.

How That One Crawler checks it

  • Blocked by robots.txtCritical

    robots.txt forbids crawlers from this URL, so search engines can't read what's on it.

    Fix: Remove or narrow the Disallow rule in robots.txt if this page should be searchable.

Frequently asked questions

Does robots.txt keep a page out of Google?

Not reliably. A blocked URL can still be indexed without its content if other sites link to it. Use noindex to keep a page out of results.

Should I block AI crawlers?

That's a business choice. Blocking them keeps your content out of AI answers, which also means AI tools can't cite or link to you.

Find every page with this problem

Free, no signup, results in a couple of minutes. Your scans stay in your browser.