That One CrawlerDaddy mode

For site owners

Meet ThatOneCrawler, the bot behind the audits

ThatOneCrawler is the user agent That One Crawler uses to read public pages when someone runs an SEO audit. It only visits when a person starts a scan, fetches a few pages at a time, never logs in or submits forms, and obeys robots.txt rules written for it.

By Updated

How it identifies itself

Requests carry this user agent:

Mozilla/5.0 (compatible; ThatOneCrawler/1.0; +https://thatonecrawler.houseofnamus.com/bot)

How it behaves

  • Visits only when someone runs a scan; there's no background crawling or indexing.
  • Fetches about six pages at a time and stops at the page limit the person chose (1,000 at most).
  • Reads your robots.txt and sitemap first, and only follows links within your site.
  • Checks outgoing links and images with lightweight HEAD requests, and only retries with a tiny GET when a server refuses HEAD.
  • Never submits forms, logs in, or fetches private or internal network addresses.
  • Results are stored in the scanning person's browser, not on our servers.

How to block or limit it

Add this to your robots.txt to keep it out entirely:

User-agent: ThatOneCrawler
Disallow: /

Or disallow just the paths you want skipped. Scans of your site will then note that some URLs were skipped at your request.

Frequently asked questions

Why did ThatOneCrawler visit my site?

Someone entered your site's address into That One Crawler to run an SEO audit. The bot never crawls on its own schedule; every visit is triggered by a person's scan.

How do I block ThatOneCrawler?

Add a group for it to your robots.txt, for example “User-agent: ThatOneCrawler” followed by “Disallow: /”. We obey rules written for our user agent, and skip the URLs they disallow.

Does it obey general robots.txt rules?

Rules for all bots (User-agent: *) are reported as findings in the audit, because site owners usually want to know which pages search engines can't reach. Rules naming ThatOneCrawler are always obeyed.