How it identifies itself
Requests carry this user agent:
Mozilla/5.0 (compatible; ThatOneCrawler/1.0; +https://thatonecrawler.houseofnamus.com/bot)
How it behaves
- Visits only when someone runs a scan; there's no background crawling or indexing.
- Fetches about six pages at a time and stops at the page limit the person chose (1,000 at most).
- Reads your robots.txt and sitemap first, and only follows links within your site.
- Checks outgoing links and images with lightweight HEAD requests, and only retries with a tiny GET when a server refuses HEAD.
- Never submits forms, logs in, or fetches private or internal network addresses.
- Results are stored in the scanning person's browser, not on our servers.
How to block or limit it
Add this to your robots.txt to keep it out entirely:
User-agent: ThatOneCrawler Disallow: /
Or disallow just the paths you want skipped. Scans of your site will then note that some URLs were skipped at your request.
Frequently asked questions
Why did ThatOneCrawler visit my site?
Someone entered your site's address into That One Crawler to run an SEO audit. The bot never crawls on its own schedule; every visit is triggered by a person's scan.
How do I block ThatOneCrawler?
Add a group for it to your robots.txt, for example “User-agent: ThatOneCrawler” followed by “Disallow: /”. We obey rules written for our user agent, and skip the URLs they disallow.
Does it obey general robots.txt rules?
Rules for all bots (User-agent: *) are reported as findings in the audit, because site owners usually want to know which pages search engines can't reach. Rules naming ThatOneCrawler are always obeyed.