latest update to awesome search engines is here:
https://github.com/davidshq/awesome-search-engines

Biggest news is I've added a page for #BuildingSearchEngines - it's very partial at the moment but includes sections on #SearchEngines (open source), #WebCrawlers, and #CommonCrawl.

Know of other web-scale search engines, crawlers, etc. I should be aware of?