Uploaded November 2026 | Updated November 2026, 4 days ago
You just added a CDN to your website to make it fast, light, and secure… and now your origin server is busier than ever? Why is sudden crawler traffic hitting your origin server like a firehose… and can blanket bot protection save the day?! 🧯🔥
Welcome to the Season 1 Finale of The Non-Canonical Cabinet, the series where Google Search Developer Advocates Gary Illyes and Cherry Prommawin dig through a (apparently, somebody else's) banged up filing cabinet to tackle the weirdest edge cases in technical SEO.
In this episode, they explain why Googlebot is firehosing your newly set up CDN, what to watch out for, and how to handle the situation calmly (while a rubber chicken dressed in bubble wrap tries desperately to solve a CAPTCHA):
⚡ Crawler Over-enthusiasm: Why crawlers ramp up crawl speed when you move to a CDN
🔥 Cache Misses & Origin Servers: Why for every unique first-time request, your origin server still does the heavy lifting anyway
🛑 Blanket Bot Protection: Does aggressively blocking all bots help? (And can crawlers solve CAPTCHAs with traffic lights or only those with chickens?)
🛡️ How to Survive CDN Launch Day: Things you can do to protect your servers from collapsing
📌 Timestamps:
00:00 – Welcome: (Why is Gary wearing a hardhat? And what's that smell?)
01:32 – Why a move to CDN can make crawlers a little too excited
02:45 – Cache misses (Why is Chicken's memory empty?)
04:10 – Origin server under siege
05:18 – Why blanket bot protection/CAPTCHAs backfire
06:33 – "Explaining networking with chickens was not in the job description"
07:15 – Best practices (or how to not set things on fire)
08:24 – Recap
09:10 – Outro: we are on fire 🔥! (Wait... whose office is this?)
📚 Learn more:
CDNs and Crawling blog post: https://goo.gle/crawling-cdn
Reduce Googlebot crawl rate: developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate
Large site crawl budget management: developers.google.com/crawling/docs/crawl-budget
🗄️ Got a weird mystery you want us to pull from the (slightly toasted) cabinet next season? Or just want to help fund our smoke machine for Season 2 (budget pending)? Drop a comment below!
#GoogleSearch #TechnicalSEO #WebDevelopment #NonCanonicalCabinet #SEO #SearchCentral #CDN #CrawlBudget
You just added a CDN to your website to make it fast, light, and secure… and now your origin server is busier than ever? Why is sudden crawler traffic hitting your origin server like a firehose… and can blanket bot protection save the day?! 🧯🔥
Welcome to the Season 1 Finale of The Non-Canonical Cabinet, the series where Google Search Developer Advocates Gary Illyes and Cherry Prommawin dig through a (apparently, somebody else's) banged up filing cabinet to tackle the weirdest edge cases in technical SEO.
In this episode, they explain why Googlebot is firehosing your newly set up CDN, what to watch out for, and how to handle the situation calmly (while a rubber chicken dressed in bubble wrap tries desperately to solve a CAPTCHA):
⚡ Crawler Over-enthusiasm: Why crawlers ramp up crawl speed when you move to a CDN
🔥 Cache Misses & Origin Servers: Why for every unique first-time request, your origin server still does the heavy lifting anyway
🛑 Blanket Bot Protection: Does aggressively blocking all bots help? (And can crawlers solve CAPTCHAs with traffic lights or only those with chickens?)
🛡️ How to Survive CDN Launch Day: Things you can do to protect your servers from collapsing
📌 Timestamps:
00:00 – Welcome: (Why is Gary wearing a hardhat? And what's that smell?)
01:32 – Why a move to CDN can make crawlers a little too excited
02:45 – Cache misses (Why is Chicken's memory empty?)
04:10 – Origin server under siege
05:18 – Why blanket bot protection/CAPTCHAs backfire
06:33 – "Explaining networking with chickens was not in the job description"
07:15 – Best practices (or how to not set things on fire)
08:24 – Recap
09:10 – Outro: we are on fire 🔥! (Wait... whose office is this?)
📚 Learn more:
CDNs and Crawling blog post: https://goo.gle/crawling-cdn
Reduce Googlebot crawl rate: developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate
Large site crawl budget management: developers.google.com/crawling/docs/crawl-budget
🗄️ Got a weird mystery you want us to pull from the (slightly toasted) cabinet next season? Or just want to help fund our smoke machine for Season 2 (budget pending)? Drop a comment below!
#GoogleSearch #TechnicalSEO #WebDevelopment #NonCanonicalCabinet #SEO #SearchCentral #CDN #CrawlBudget










