# Meet ThatOneCrawler, the bot behind the audits

> ThatOneCrawler visits pages only when someone runs an SEO scan. Learn how it behaves, how gently it crawls, and how to block it with robots.txt.

Source: https://thatonecrawler.houseofnamus.com/bot · Updated 2026-10-22 · By Suman Debnath (https://sumandebnath.houseofnamus.com)

ThatOneCrawler is the user agent That One Crawler uses to read public pages when someone runs an SEO audit. It only visits when a person starts a scan, fetches a few pages at a time, never logs in or submits forms, and obeys robots.txt rules written for it.

## How it identifies itself

Requests carry this user agent:

```
Mozilla/5.0 (compatible; ThatOneCrawler/1.0; +https://thatonecrawler.houseofnamus.com/bot)
```

## How it behaves

- Visits only when someone runs a scan; there's no background crawling or indexing.
- Fetches about six pages at a time and stops at the page limit the person chose (1,000 at most).
- Reads your robots.txt and sitemap first, and only follows links within your site.
- Checks outgoing links and images with lightweight HEAD requests, and only retries with a tiny GET when a server refuses HEAD.
- Never submits forms, logs in, or fetches private or internal network addresses.
- Results are stored in the scanning person's browser, not on our servers.

## How to block or limit it

Add this to your robots.txt to keep it out entirely:

```
User-agent: ThatOneCrawler
Disallow: /
```

Or disallow just the paths you want skipped.

## Frequently asked questions

### Why did ThatOneCrawler visit my site?

Someone entered your site's address into That One Crawler to run an SEO audit. The bot never crawls on its own schedule; every visit is triggered by a person's scan.

### How do I block ThatOneCrawler?

Add a group for it to your robots.txt, for example “User-agent: ThatOneCrawler” followed by “Disallow: /”. We obey rules written for our user agent, and skip the URLs they disallow.

### Does it obey general robots.txt rules?

Rules for all bots (User-agent: *) are reported as findings in the audit, because site owners usually want to know which pages search engines can't reach. Rules naming ThatOneCrawler are always obeyed.

---

That One Crawler: Free SEO audit tool for blogs. Crawl your whole site to find broken links, thin content, duplicate titles, keyword cannibalization and AI search gaps. Run a free audit at https://thatonecrawler.houseofnamus.com. All pages: https://thatonecrawler.houseofnamus.com/llms.txt
