Technical SEO Tip: The New York Times uses a "Today" sitemap to link to their most recent articles. This strategy ensures that Googlebot always has a path to crawl their highest priority content:

#newyorktimes #seotips #sitemap #seostrategy #googlebot #technicalseo #crawling #content #contentstrategy

It's quite impressive to have upgraded OTF and the site immediately get slammed by 6,000 #Googlebot requests in two minutes. And when I go to try and stop it, I can't. And then it starts telling me about all the "faults" on the site. And I can't limit the crawl rate, I have to "make a special request".

Fuck off #Google. It's *my* site, not yours to hammer and critique as you wish.

Validate Googlebot IP addresses using API for in Excel spreadsheets, Python code, cURL, etc.

https://seoapi.com/googlebot/

#seo #googlebot #google #search #seotip #excel

Bulk generate non-existing URLs on a competitor's site that lead to 5XX server errors when opened, e.g. such as this:

https://example.com/fold/做假巴基斯坦伊斯兰堡护照-做假巴基斯坦伊斯兰堡护照-(⏩办证网址✔️ikmm8839.com⏪)-加拿大汉密尔顿做假护照-做假巴基斯坦伊斯兰堡护照-做假巴基斯坦伊斯兰堡护照-(⏩办证网址)

Help #googlebot visit these URLs.

Googlebot sees that a substantial number of pages on that domain return 5XX, the server is unable to handle requests.

#Google reduces the page #crawl frequency for that domain.

I saw some signs of this the other day. #seo

I came back to my desk in the office to find a Gaggle of Googlebots messing around. These designs have evolved over the years; I haven't had a chance to try the newer #Googlebot variations though.

Given their order, I guess you could say size is a ranking factor? #SEO

The story of Googlebot and the cats ...

#seo #googlebot #cats #comic #japanese

Using Cloudflare?

Then don't forget to add "Disallow: /cdn-cgi/" to your robots.txt, as you don't want #GoogleBot to spend resources on crawling unnecessary files.

It's also recommended in their official docs: https://developers.cloudflare.com/fundamentals/get-started/reference/cdn-cgi-endpoint/

Un article qui partage une vision similaire à la mienne sur la façon d'optimiser le crawl de Googlebot sur les pages de votre qui sont importantes.

https://searchengineland.com/crawl-efficacy-optimization-389085

#seo #crawl #google #googlebot

Omacronides 12 : Googlebot, javascript, indexation et site web statique <https://omacronides.com/notes/2022-09-30/>

Comment passer son site web en full javascript détruit le référencement des pages, et comment construite un système hybride alliant full javascript et pages html statiques.

#javascript #html #googlebot