Many of the biggest "open AI" companies are totally opaque when it comes to training data. Google and OpenAI won't even say how many pieces of data went into their models' training - let alone which data they used.

Other "open AI" companies use publicly available datasets like #ThePile and #CommonCrawl. But you can't replicate their models by shoveling these datasets into an algorithm. Each one has to be groomed - labeled, sorted, de-duplicated, and otherwise filtered.

28/

when #AWS is rate limiting the #CommonCrawl dataset and you are trying to use it at a #hackathon. 😭

@jrp @sl007 Da es sich beim #CommonCrawl um #OpenData handelt, würde ich seinen Crawler nicht aussperren.

Aus eigener Erfahrung kann ich sagen, dass man gleichzeitig der #OpenEverything- und der #Privacy-Bewegung angehören kann.

@ShaulaEvans I got curious about this and just learned you can block the user agent “CCBot” in robots.txt to block #CommonCrawl, the crawler whose corpus (as far as I can tell) is used by #OpenAI and Google #Bard.

latest update to awesome search engines is here:
https://github.com/davidshq/awesome-search-engines

Biggest news is I've added a page for #BuildingSearchEngines - it's very partial at the moment but includes sections on #SearchEngines (open source), #WebCrawlers, and #CommonCrawl.

Know of other web-scale search engines, crawlers, etc. I should be aware of?

Current personal principles update:

1. All public data should be libre.
2. Fediverse posts seem to be public.

Obviously, I can't enforce my principles on others, only myself. Therefore:

1. My toots are now CC-BY-SA 4.0.
2. I no longer opt-out of indexing.
3. I'll resume my use of search engines. Unfortunately, they're all closed-source. #CommonCrawl?
4. I'll now use AI tools (eg art and code generation), as long as they're libre software.

I may change my opinions again.