OmniCheck's crawler

OmniCheck is a website health checker. It runs accessibility, broken-link, performance and AI-readiness checks against a website and returns a report to the person who asked for it.

It is not a search engine, an archiver, or a training-data collector. It does not index your content, retain page bodies beyond producing a report, or pass what it fetches to anyone other than the person who ran the scan.

Every request OmniCheck makes is because a person asked for it. There is no continuous crawl and no discovery of sites on our own initiative. Traffic you see from us is one of two things:

• A scan someone submitted. Somebody entered your URL and asked for a report. That may be you, someone at your organisation, an agency, or a person evaluating the site.
• Scheduled monitoring. An OmniCheck account holder added the site to monitoring, and we re-check it on the interval they chose.

We do not verify that the person submitting a URL owns the site. If OmniCheck traffic is unwelcome, the sections below tell you how to stop it, and blocking us does not require contacting us first.

OmniCheck requests arrive with one of two user agents, and it is worth knowing about both.

When a site accepts an identified crawler, we identify:

OmniCheckBot/1.0 (+https://omnicheck.site/bot)

By default, however, requests carry an ordinary browser user agent:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36

We would rather always use the first one, and we are telling you about the second rather than letting you discover it. The reason is practical: a meaningful number of firewalls refuse anything that identifies itself as a crawler, and several refuse it by matching the word "scanner" in the user agent alone. A site that refuses the identified request is not usually making a considered decision about OmniCheck — it is applying a blanket rule, and the result is a report that says the site has no pages. So the scanner leads with a browser user agent and switches to the identified one when a site turns it away.

That trade disappears the moment you allowlist us. If OmniCheck is permitted on your site, it will identify consistently as OmniCheckBot, which makes it easy for you to see in logs and easy to scope firewall rules against. See how to allow us.

We do not use residential proxies, rotate through anonymising networks, or attempt to defeat CAPTCHAs or JavaScript challenges. If a challenge is served, the scan records that it was refused and stops.

A scan is bounded and short-lived. It is not a crawl of your whole site.

• A preview scan fetches at most 3 pages.
• A full scan fetches at most 50 pages for ordinary accounts.
• We prefer your sitemap and only crawl links when no sitemap is readable.
• Large sites are sampled by page template rather than fetched exhaustively.

If OmniCheck is causing load problems on your site, tell us and we will look at it — see contact.

We want to be exact here rather than make a blanket claim, because the answer differs by check:

• The broken-link crawler honours robots.txt. This is on by default; the person running the scan can turn it off for a site they control.
• Page discovery reads robots.txt only for the Sitemap: directive, and does not currently apply Disallow rules when falling back to crawling links. We are aware of the gap and intend to close it.
• The AI-readiness check reads robots.txt to report on it — which AI crawlers you allow or block is part of what that report tells you.

Because of the second point, robots.txt alone is not currently a reliable way to keep OmniCheck off a page. If you need us stopped, use a method from the next section.

You are entitled to refuse automated traffic, and you do not owe us a reason. Any of these will work:

• Block the user agent at your firewall or CDN. Rules matching OmniCheckBot will catch our identified requests.
• Existing bot protection. Cloudflare, SiteGround, Sucuri, Wordfence and similar products already stop us when configured to challenge automated traffic; no OmniCheck-specific rule is needed.
• Ask us. Email the address below and we will add your domain to a refusal list.

A blocked scan fails with a message telling the person who ran it that the site declined automated access. It does not report your site as broken, and it does not count against any score.

If you are monitoring your own site with OmniCheck, allowlisting makes results reliable and stops your firewall spending effort on us.

The most precise method is to match on the user agent:

User-Agent contains "OmniCheckBot"

In Cloudflare this is a WAF skip rule; in SiteGround, Wordfence and Sucuri it is an allowlist entry. Scope it as narrowly as your tooling permits.

A machine-readable summary is published at https://omnicheck.site/bot/ips.json.

We do not currently publish IP ranges, and the list at that URL is deliberately empty. OmniCheck scans presently leave from the same address as unrelated services, and telling you to allowlist it would be telling you to allowlist traffic that is not ours. When scanning has an address of its own, it will be published there and this page will say so. Until then, allowlist on the user agent.

For anything about this crawler — load problems, a block request, or a question this page does not answer — email [email protected]. Include the domain and roughly when you saw the traffic, and we will be able to find it.