Uploaded December 2024 | Updated September 2026, 2 weeks ago

---

Introduction to robots.txt → https://goo.gle/4gbNmcl
Control what you share with Google → https://goo.gle/3VnyLBU
Open Source robotstxt library & tester → github.com/google/robotstxt

What is a robots.txt file and how does it impact website indexability? Join Martin Splitt on this Search Lightning Talk as he covers how robots.txt interacts with robots meta tags, HTTP headers, and how to use them to control search engine crawlers' access to your website.

Chapters:
0:00 - Intro
0:17 - Why do I need robots.txt?
0:40 - What is the robots meta tag?
2:47 - When to use noindex vs. disallow
3:48 - Why is Googlebot still crawling certain pages?
5:14 - Conclusion

Watch more Search Central Lightning Talks → https://goo.gle/lightning-talks
Subscribe to the Google Search Central YouTube Channel → https://goo.gle/SearchCentral

#GoogleSearch #SearchLightningTalks

Speaker: Martin Splitt
Products Mentioned: Search Console