Uploaded January 2023 | Updated September 2026, 2 days ago
Abstract:
I will present the results of the NLP Community Metasurvey (nlpsurvey.net). This was a questionnaire that we ran from May to June 2022 which elicited the opinions of NLP researchers on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies. For example, respondents are split almost exactly in half on questions about:
* the importance of artificial general intelligence,
* whether language models understand language, and
* the necessity of linguistic structure and inductive bias for solving NLP problems.
In addition, the survey posed "meta-questions," asking respondents to predict the distribution of survey responses. This allows us not only to gain insight on the spectrum of beliefs held by NLP researchers, but also to uncover false sociological beliefs where the community’s predictions don’t match reality. We find such mismatches on a wide range of issues. Among other results, the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its own belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.
Our hope is that this can provide context for the NLP research community to have more informed and self-aware discussions of these complex issues. In this talk, I will walk through our results and open the floor for such a discussion.
Bio:
Julian Michael (julianmichael.org) is a postdoc at New York University working with Sam Bowman in the Alignment Research Group. His current research is on scalable supervision to extract truth and scientific insight from highly-capable machine learning models. In 2022, he finished his PhD with Luke Zettlemoyer at the University of Washington, where he developed approaches for annotation and manipulation of English semantic structure, as well as working on benchmarking, explainability, model analysis, formal logic, language model prompting, and question answering and generation. His work has appeared in conferences including ACL, EMNLP, NAACL, NeurIPS, and ICLR.
Abstract:
I will present the results of the NLP Community Metasurvey (nlpsurvey.net). This was a questionnaire that we ran from May to June 2022 which elicited the opinions of NLP researchers on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies. For example, respondents are split almost exactly in half on questions about:
* the importance of artificial general intelligence,
* whether language models understand language, and
* the necessity of linguistic structure and inductive bias for solving NLP problems.
In addition, the survey posed "meta-questions," asking respondents to predict the distribution of survey responses. This allows us not only to gain insight on the spectrum of beliefs held by NLP researchers, but also to uncover false sociological beliefs where the community’s predictions don’t match reality. We find such mismatches on a wide range of issues. Among other results, the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its own belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.
Our hope is that this can provide context for the NLP research community to have more informed and self-aware discussions of these complex issues. In this talk, I will walk through our results and open the floor for such a discussion.
Bio:
Julian Michael (julianmichael.org) is a postdoc at New York University working with Sam Bowman in the Alignment Research Group. His current research is on scalable supervision to extract truth and scientific insight from highly-capable machine learning models. In 2022, he finished his PhD with Luke Zettlemoyer at the University of Washington, where he developed approaches for annotation and manipulation of English semantic structure, as well as working on benchmarking, explainability, model analysis, formal logic, language model prompting, and question answering and generation. His work has appeared in conferences including ACL, EMNLP, NAACL, NeurIPS, and ICLR.
