Googlebot May Be Hitting Bot Blocker URLs

Many sites use larva blockers like Distil Networks to prevent scrapers from stealing their knowledge. The challenge for SEOs is that generally these larva blockers aren’t set properly and may forestall sensible bots like Googlebot and BingBot from aiming to the content, which might cause serious SEO problems. Distil is pretty adamant that their service is SEO-safe, however i’m not therefore sure concerning others. we have a tendency to recently saw a case wherever the communications protocol version of a giant site’s homepage was causation Googlebot to a 404 URL whereas causation users to the https homepage, all as a result of the larva blocker (not Distil) wasn’t tuned properly. Was quite funny finding out their complete and generally seeing “Oops” because the 1st result. It wasn’t that funny for the shopper.

So after we see odd travel behavior or alternative odd signals, we have a tendency to at once begin questioning if it might be the larva blocker.

This we have a tendency toek we saw URLs like /distil_r_blocked.html?… begin to point out up during a client’s GSC crawl errors report below the opposite tab. I at once went into holy larva blocker Batman! mode. you certainly don’t wish Googlebot touching larva blocker URLs.

It looks as if these guys tried to scrape the location, hit Distil URLs and so revealed them as links. therefore once Googlebot crawled them (why Googlebot would wish to crawl these crap sites is another issue) it’d hit the Distil URLs.

I am still extremely suspicious of larva blockers once it involves SEO, due to their under-the-hood nature, however next time you see Distil URLs show up in GSC, don’t panic. it should simply be those crap links your website is therefore sensible at attracting.

Add a Comment

Your email address will not be published. Required fields are marked *