# Blocked by robots.txt: what it means and how to fix it

> Pages blocked by robots.txt can't be read by search engines. Learn how to spot rules that block important pages and how to edit your robots.txt file safely.

Source: https://thatonecrawler.houseofnamus.com/checks/blocked-by-robots-txt · Updated 2026-10-22 · By Suman Debnath (https://sumandebnath.houseofnamus.com)

“Blocked by robots.txt” means a Disallow rule in your site's robots.txt file stops crawlers from fetching that URL. Search engines can't read what's on the page, so it ranks poorly or not at all. Remove or narrow the rule if the page should be searchable.

## Why it matters

robots.txt rules are powerful and easy to over-apply: one line like “Disallow: /blog” hides a whole section. Rules copied from another site or left over from development are common culprits. Blocking also stops search engines from seeing a noindex tag on the page.

## How to fix it

1. Open yoursite.com/robots.txt and read the rules under “User-agent: *”.
2. Find the Disallow line matching the blocked URL (the longest matching rule wins).
3. Delete it, or narrow it so it only covers what should stay private.
4. Keep admin, cart and internal search pages blocked if you like: they don't need to rank.
5. Rescan to confirm important pages are no longer blocked.

## Frequently asked questions

### Does robots.txt keep a page out of Google?

Not reliably. A blocked URL can still be indexed without its content if other sites link to it. Use noindex to keep a page out of results.

### Should I block AI crawlers?

That's a business choice. Blocking them keeps your content out of AI answers, which also means AI tools can't cite or link to you.

## Related checks

- [Noindex tag: why your page isn't in Google and how to fix it](https://thatonecrawler.houseofnamus.com/checks/noindex-tag)
- [AI search readiness: how to get your blog cited by AI (GEO and AEO)](https://thatonecrawler.houseofnamus.com/checks/ai-search-readiness)

---

That One Crawler: Free SEO audit tool for blogs. Crawl your whole site to find broken links, thin content, duplicate titles, keyword cannibalization and AI search gaps. Run a free audit at https://thatonecrawler.houseofnamus.com. All pages: https://thatonecrawler.houseofnamus.com/llms.txt
