Total pages are blocked.

[kexej28769@nongnue]

According to our research, Majestic ended up with 17,787,118 referring domains, Ahrefs with 20,072,690, and Moz with 16,598,365. Again, Moz’s robots.txt profile was similar to Google’s. But referring domains aren’t the only issue we need to be concerned with.


Most pages on the web only have internal links. Google isn’t interested in building a link graph — they’re interested in building a search engine. So, a bot designed to work like Google needs to worry about pages that only get internal links as much as they do those that get external links. Another metric we can measure is the total number of pages that are blocked using Google’s site: a query to estimate the number of pages that Google has access to that no other crawler has. So, how do competitive industry crawlers perform? Lower is better.


Once again, Moz shines on this metric. It’s not just that benin number data is blocked by fewer sites — Moz is blocked by less important and smaller sites. Majestic loses 675,381,982 pages, Ahrefs loses 732,871,714, and Moz loses 658,015,885. There’s a difference of about 80 million pages between Ahrefs and Moz in just the top million sites on the web.

Unique sites were blocked.
Most robots.txt disapprovals faced by Moz, Majestic, and Ahrefs are blanket blocks of all bots that do not represent the major search engines. However, we can isolate times when specific bots are intentionally designated for exclusion while competitors remain. For example, how often does Moz get blocked while Ahrefs and