I can't crawl the site with my SEO Bot (Moz/Ahrefs etc)
As a security measure, our servers by default block a bunch of bots as follows:
- ahrefs
- alexibot
- majestic
- mj12bot
- rogerbot
This is because they can be faked fairly easily meaning they could be malicious. Secondly when the legitimate tool is used it can result in severe performance complications for your website. These SEO cralwers and bots are not well known for being moderate in their crawl rates.
If you wish to have these bots crawl your site, please open a support ticket and we'll make those arrangements.
The blocking could be happening at the cloudflare WAF firewall, or in the on server security, so if you are able to be specific about when you requested the crawl, and the bot you are tryig to use that would be useful.
