Dave Mackey · @davidshq
802 followers · 1237 posts · Server hachyderm.ioWhat are your favorite / the best #WebCrawlers for broad / #WebScale #crawling?
I've built a list but am looking for anything I missed: https://github.com/davidshq/awesome-search-engines/blob/main/WebCrawlers.md
Main options I've found include #Apache #Nutch, #StormCrawler, #Scrapy, #Norconex, #PulsarR, #Heritrix, and #sparkler
#WebCrawlers #webscale #crawling #apache #nutch #stormcrawler #scrapy #norconex #pulsarr #heritrix #sparkler #question #search #searchengines
Dave Mackey · @davidshq
685 followers · 970 posts · Server hachyderm.io
latest update to awesome search engines is here:
https://github.com/davidshq/awesome-search-engines
Biggest news is I've added a page for #BuildingSearchEngines - it's very partial at the moment but includes sections on #SearchEngines (open source), #WebCrawlers, and #CommonCrawl.
Know of other web-scale search engines, crawlers, etc. I should be aware of?
#buildingsearchengines #searchengines #WebCrawlers #commoncrawl
Volkan Özçelik · @volkan
16 followers · 821 posts · Server z2h.dev
Katana is a crawling and spidering framework
#WebCrawlers #tools #scrapers #apps