Uploaded July 2024 | Updated September 2026, 2 hours ago
The cognitive reflection test was meant to measure one thing: the tendency for someone to stop and think before giving a quick answer. The simplicity of the test made it a widely used measure for critical thinking. But some clever experiments illustrate that the CRT is not so simple after all.
00:00 Introduction
00:19 The CRT problems
1:20 The structure of CRT problems
3:04 Some hints about what's going wrong
4:53 The structure of insight problems
6:27 Explanation Number One
7:57 Explanation Number Two
8:57 Explanation Number Three
Join my online learning community: benjaminkeep.com/community
REFERENCES
This video is mostly based off of Patel, N., Baker, S. G., & Scherer, L. D. (2019). Evaluating the cognitive reflection test as a measure of intuition/reflection, numeracy, and insight problem solving, and the implications for understanding real-world judgments and beliefs. Journal of Experimental Psychology: General, 148(12), 2129–2153. doi.org/10.1037/xge0000592
This is the (classic) paper that introduced the CRT as a measure of reflection. Frederick, S. (2005). Cognitive reflection and decision making. The Journal of Economic Perspectives, 19, 25– 42. dx.doi.org/10.1257 089533005775196732. It also includes correlations between the math section of the SAT and the CRT (.46) vs the verbal section of the SAT and the CRT (.24).
On the CRT’s correlation to tests measuring numeracy being about .5, see Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia-Retamero, R. (2012). Measuring Risk Literacy: The Berlin Numeracy Test. Judgment and Decision Making, 7(1), 25–47. doi.org/10.1017/S1930297500001819
On the associations between reflection/analytical thinking and various beliefs, see Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015). Everyday Consequences of Analytic Thinking. Current Directions in Psychological Science, 24(6), 425-432. doi.org/10.1177/0963721415604610
This mentions some of the earlier results correlating CRT performance to rational thinking, but also extends the CRT to a seven-question test. The four added questions are probably even more reliant on numerical ability than the first three. Toplak, M. E., West, R. F., & Stanovich, K. E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & cognition, 39(7), 1275-1289.
The cognitive reflection test was meant to measure one thing: the tendency for someone to stop and think before giving a quick answer. The simplicity of the test made it a widely used measure for critical thinking. But some clever experiments illustrate that the CRT is not so simple after all.
00:00 Introduction
00:19 The CRT problems
1:20 The structure of CRT problems
3:04 Some hints about what's going wrong
4:53 The structure of insight problems
6:27 Explanation Number One
7:57 Explanation Number Two
8:57 Explanation Number Three
Join my online learning community: benjaminkeep.com/community
REFERENCES
This video is mostly based off of Patel, N., Baker, S. G., & Scherer, L. D. (2019). Evaluating the cognitive reflection test as a measure of intuition/reflection, numeracy, and insight problem solving, and the implications for understanding real-world judgments and beliefs. Journal of Experimental Psychology: General, 148(12), 2129–2153. doi.org/10.1037/xge0000592
This is the (classic) paper that introduced the CRT as a measure of reflection. Frederick, S. (2005). Cognitive reflection and decision making. The Journal of Economic Perspectives, 19, 25– 42. dx.doi.org/10.1257 089533005775196732. It also includes correlations between the math section of the SAT and the CRT (.46) vs the verbal section of the SAT and the CRT (.24).
On the CRT’s correlation to tests measuring numeracy being about .5, see Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia-Retamero, R. (2012). Measuring Risk Literacy: The Berlin Numeracy Test. Judgment and Decision Making, 7(1), 25–47. doi.org/10.1017/S1930297500001819
On the associations between reflection/analytical thinking and various beliefs, see Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015). Everyday Consequences of Analytic Thinking. Current Directions in Psychological Science, 24(6), 425-432. doi.org/10.1177/0963721415604610
This mentions some of the earlier results correlating CRT performance to rational thinking, but also extends the CRT to a seven-question test. The four added questions are probably even more reliant on numerical ability than the first three. Toplak, M. E., West, R. F., & Stanovich, K. E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & cognition, 39(7), 1275-1289.
