Read all about the OWLer, the OpenWebSearch.EU crawler:
https://openwebsearch.eu/owler/
We crawl the Web to create an open web index and to bootstrap a more open internet search ecosystem, which also includes that webmasters and content producers have more control over how and for what their precious content is used.
See also: @osf