HAI Seminar: Addressing Challenges of Public Web Data @stanfordhai
HAI Seminar: Addressing Challenges of Public Web Data  @stanfordhai
Uploaded October 2025 | Updated November 2026, 3 hours ago
This HAI seminar featured Common Crawl Foundation’s work on preserving humanity's knowledge and making it accessible through its free public web dataset, a vital resource since 2008. The Common Crawl team presented insights from a new data product that utilizes Common Crawl's metadata to explore concerns around robots.txt exclusions, legal demands, and "bot defenses," advocating for greater transparency and informed solutions for the future of public web data.

This seminar was recorded on October 22, 2025 at Stanford University.

00:00:00 Introduction
00:01:01 Lecture
00:48:53 Q&A
HAI Seminar: Addressing Challenges of Public Web DataHAI at Five Conference: Godmothers of AIThe Role of Business: Policy Implications of Industry Leadership in AIHAI Seminar: Melissa Valentine and Michael Bernstein on Flash TeamsInside Stanford HAIs Congressional Boot Camp on AIAI+Science: AI for the Universe2026 AI for Mental Health (AI4MH) Symposium: Industry & Translation —What It Takes to Deploy2026 Conference on Physics and AI: Risa WechslerBuilding an International AI Platform for Health Professions Education and the Learning Sciences2026 Conference on Physics and AI: Panel 1Confronting Our AI Future: Why Universities Matter for AI with Stanford Provost Jenny MartinezCharting the Future: Open Innovation in an Era of Global AI Competition
Stanford HAI |

HAI Seminar: Addressing Challenges of Public Web Data

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER