Why it matters
robots.txt rules are powerful and easy to over-apply: one line like “Disallow: /blog” hides a whole section. Rules copied from another site or left over from development are common culprits. Blocking also stops search engines from seeing a noindex tag on the page.
How to fix it
Open yoursite.com/robots.txt and read the rules under “User-agent: *”.
Find the Disallow line matching the blocked URL (the longest matching rule wins).
Delete it, or narrow it so it only covers what should stay private.
Keep admin, cart and internal search pages blocked if you like: they don't need to rank.
Rescan to confirm important pages are no longer blocked.
How That One Crawler checks it
Blocked by robots.txtCritical
robots.txt forbids crawlers from this URL, so search engines can't read what's on it.
Fix: Remove or narrow the Disallow rule in robots.txt if this page should be searchable.
Frequently asked questions
Does robots.txt keep a page out of Google?
Not reliably. A blocked URL can still be indexed without its content if other sites link to it. Use noindex to keep a page out of results.
Should I block AI crawlers?
That's a business choice. Blocking them keeps your content out of AI answers, which also means AI tools can't cite or link to you.
Find every page with this problem
Free, no signup, results in a couple of minutes. Your scans stay in your browser.