Uploaded October 2020 | Updated September 2026, 7 hours ago
Education researchers have traditionally faced severe data limitations in studying local policy variation; administrative datasets capture only a fraction of districts’ policy decisions, and it can be expensive to collect more nuanced implementation data from teachers and leaders. Natural language processing and web-scraping techniques can help address these challenges by assisting researchers in locating and processing policy documents located online. School district policies and practices are commonly documented in student and staff manuals, school improvement plans, and meeting minutes that are posted for the public. This paper introduces an end-to-end framework for collecting these sorts of policy documents and extracting structured policy data: the researcher gathers all potentially relevant documents from district websites, narrows the text corpus to spans of interest using a text classifier, and then extracts specific policy data using additional natural language processing techniques. Through this framework, a researcher can describe variation in policy implementation at the local level, aggregated across the state- or nation-wide populations even as policies evolve over time.
Education researchers have traditionally faced severe data limitations in studying local policy variation; administrative datasets capture only a fraction of districts’ policy decisions, and it can be expensive to collect more nuanced implementation data from teachers and leaders. Natural language processing and web-scraping techniques can help address these challenges by assisting researchers in locating and processing policy documents located online. School district policies and practices are commonly documented in student and staff manuals, school improvement plans, and meeting minutes that are posted for the public. This paper introduces an end-to-end framework for collecting these sorts of policy documents and extracting structured policy data: the researcher gathers all potentially relevant documents from district websites, narrows the text corpus to spans of interest using a text classifier, and then extracts specific policy data using additional natural language processing techniques. Through this framework, a researcher can describe variation in policy implementation at the local level, aggregated across the state- or nation-wide populations even as policies evolve over time.
