Unsurprisingly, webmeup's assurance that "you will not see recurring requests from the BLEXBot crawler to the same page" turns out to be ... not true?

At least according to my log files, which show the same page getting hit at 5 day intervals as part of their process of fetching every single page on my site over and over to satisfy some vague marketing need.

So I think BLEXBot can join AHRefsBot and SEMRushBot in my robots.txt. And nothing of value was lost.

#crawlers #webspiders #robotstxt