PriEco · @prieco
9 followers · 26 posts · Server fosstodon.org
PriEco · @prieco
7 followers · 22 posts · Server fosstodon.orgIf somebody is interested, what efficiency PriEco's crawler now has:
It is 250k websites per day. Really want to increase it to 800k per day, maybe tomorrow.
#prieco #search #opensource #work #programming #code #crawler
#prieco #search #opensource #work #programming #code #crawler
PriEco · @prieco
7 followers · 20 posts · Server fosstodon.org😴 Working and working, trying to create the web crawler that crawls at least 800k websites per day.
Still not there yet, but getting closer
#prieco #search #opensource #work #programming #code #crawler
#prieco #search #opensource #work #programming #code #crawler
PrivacyDigest · @PrivacyDigest
541 followers · 2002 posts · Server mas.to
einfachnurRoland · @einfachnurRoland
41 followers · 1869 posts · Server nrw.social@geropflueger Inwieweit kann das funktionieren?
Die #robots.txt ist doch etwas, das vom #Crawler ausgewertet werden muss, oder nicht? Wer zwingt ihn dazu das zu tun?
Wenn ich meinen Content als Seitenbetreiber schützen will, werde ich das doch nur aktiv tun können. Gibt es da keine Möglichkeit einen Blocker zu bauen? Ich stelle fest, das kommt jemand vorbei, dem ich nicht traue und präsentiere ihm einen speziellen Content. Soll die #KI doch aus Blank pages, random-Texten oder Dickpics lernen.
flumen_calculi · @flumen_calculi
306 followers · 16677 posts · Server ruhr.socialGPTbot per robots.txt aussperren
https://platform.openai.com/docs/gptbot
(via http://www.tamagothi.de/2023/08/07/frisch-in-der-robots-txt/)
#chatgpt #urheberrecht #crawler
Elias Dabbas :verified: · @elias
54 followers · 88 posts · Server seocommunity.social🕸🕷🕸🕷🕸🕷🕸
#advertools +
@JupyterNaas
= Cloud #SEO #Crawler
🔵 Low code
🔵 Save crawl templates to re-run multiple times
🔵 Create a separate template for each website
🔵 Run multiple crawls at the same time
🔵 Enjoy!
#advertools #seo #crawler #datascience #python #DigitalMarketing #digitalanalytics
TikiBunnyImports 🍹🐇🛻🏳️🌈 · @tikibunnyimports
24 followers · 29 posts · Server weird.autos
Eric Mächler · @eric_maechler
197 followers · 957 posts · Server mastodon.digitalsuccess.dev
Heron&Fox Photo · @heronfoxphoto
30 followers · 228 posts · Server universeodon.com
A #photograph of the lower section of an #Orion #crew #capsule in a work platform at the old “Manned Spaceflight Operations Center” at #KSC, now the #NeilArmstrong Operations and Checkout Building.
The new Orion integration pathway allows the Orion #spacecraft to be assembled, stacked on the #EuropeanServiceModule, and readied for #flight atop an #SLS #rocket.
The #LaunchAbortSystem integrated in a separate facility since the #LAS contains solid fueled rockets already. The stack is then integrated on top of SLS inside the Vehicle Assembly Building #VAB and then rolled out to #LC39B on the #Crawler-Transporter for #launch.
The O&C building, VAB, Crawler-Transporter, and LC-29B #infrastructure all have #Apollo-era heritage.
https://heronfox.pixels.com/featured/orion-capsule-bottom-heron-and-fox.html
#photograph #orion #crew #capsule #ksc #neilarmstrong #spacecraft #europeanservicemodule #flight #sls #rocket #launchabortsystem #las #vab #lc39b #crawler #launch #infrastructure #apollo
michabbb · @michabbb
14 followers · 206 posts · Server social.vivaldi.net
SΤΣΡHΔΠ · @backlogmann
12 followers · 263 posts · Server mastodon.social
SΤΣΡHΔΠ · @backlogmann
12 followers · 263 posts · Server mastodon.social
SΤΣΡHΔΠ · @backlogmann
12 followers · 263 posts · Server mastodon.social
Anscheinend akzeptieren einige KIs (z.B. DevianArt DreamUp) auch spezielle meta robots Angaben wie "noai,noimageai".
Leider konnte ich bisher keine Infos dazu finden, ob z.B. OpenAIs Crawler für ChatGPT / Dall-E diese meta robots verstehen und verarbeiten können. Weiß das jemand? :welp:
#dreamup #noai noimageai #blockai #openai #chatgpt #crawler #bookmark #fueraufmklo
https://www.aimeecozza.com/noai-noimageai-meta-tag-how-to-install/
#dreamUp #noai #blockai #openai #chatgpt #crawler #bookmark #fueraufmklo
SΤΣΡHΔΠ · @backlogmann
12 followers · 263 posts · Server mastodon.social
Ein interessanter Artikel zu dem Thema. 🤔
#blockai #openai #chatgpt #crawler #bookmark #fueraufmklo
https://www.searchenginejournal.com/how-to-block-chatgpt-from-using-your-website-content/478384/
#blockai #openai #chatgpt #crawler #bookmark #fueraufmklo
Sebastian Meineck · @sebmeineck
4545 followers · 277 posts · Server mastodon.social
#Daten suchen, #Crawler basteln, Thesen schärfen. Im Interview für den neuen Online-Recherche #Newsletter führt Niclas Bodenmann hinter die Kulissen einer datenjournalistischen Recherche. Für den SRF hat er hasserfüllte Amazon-Rezensionen zum Roman #Blutbuch untersucht. Welche Werkzeuge er nutzte und warum manches im Papierkorb landete, berichtet Niclas hier:
📝 lesen https://ornarchiv.wordpress.com/2023/02/06/interview-review-bombing-auf-amazon-durchleuchten/
📯 abonnieren https://newsletter.sebmeineck.de/home
#osint #daten #crawler #newsletter #blutbuch
Destiny Hailstorm · @DestinyHailstorm
34 followers · 201 posts · Server social.transrights.artHey #Mastodon ! Specifically looking for you #software #developer peeps.
I'm working on a search engine project, and wondering if there is a general code of ethics for #web #crawler s? Or things to pay attention to websites are signaling and how I should handle them? Maybe just general tips XD
I could very easily accidentally DDOS smaller sites in my quest to index the internet for fun (or incinerate my #PiHole server), so I want to make sure I'm doing the right thing!
Any ideas?
#mastodon #software #developer #web #crawler #pihole
José Pedro Mayo · @jpmayo
7 followers · 22 posts · Server infosec.exchangeoffsec.tools - A vast collection of security tools
#CyberSecurity #osint #pentest #scanner #cve #vulnerabilities #burpsuite #endpoints #passwords #cloud #secrets #fuzzing #dns #ips #framework #network #directories #crawler #screeenshots #git #cms #allinone #proxy #probing
#cybersecurity #osint #pentest #scanner #cve #vulnerabilities #burpsuite #endpoints #passwords #cloud #secrets #fuzzing #dns #ips #framework #network #directories #crawler #screeenshots #git #cms #allinone #proxy #probing
Kyle Kurth · @2KKyle
5 followers · 26 posts · Server twit.social#Redcat let us see the #Gen9 #Crawler today! It looks awesome. Besides enhancements to the front axle and two speed transmission it is pretty similar to the #Gen8. My favorite part is the #Scout800 body with normal looking wheel openings instead of the bulky flares on the #ScoutII
#redcat #gen9 #crawler #gen8 #scout800 #scoutii
