Ross A. Baker · @ross
838 followers · 964 posts · Server social.rossabaker.comSetting up /robots.txt, not because it helps, but because being crabby in compliance with an RFC is satisfying.
Who has some unsavory ones besides ChatGPT and Twitterbot?
Last updated 1 year ago
Sebastian Nagel · @sebnagel
4 followers · 4 posts · Server fosstodon.orgReleased #CrawlerCommons 1.4: Java 11, #RobotsTxt compliant with #rfc9309 - https://github.com/crawler-commons/crawler-commons#18th-july-2023----crawler-commons-14-released
#crawlercommons #robotstxt #rfc9309
Last updated 1 year ago