Benjamin Keep, PhD, JD | How to Read Peer-reviewed Articles on Experiments in Learning @benjaminkeep | Uploaded May 2023 | Updated October 2024, 7 hours ago.
Reading peer-reviewed social science research can be quite challenging. But it’s a skill that lets you go beyond media reports of the latest study. Here’s 15 minutes of advice on how to do read experimental studies specifically.
0:00 An intro to reading social science research
0:39 What do learning researchers study?
1:54 The basics of experimental designs
5:01 What’s the goal of reading research?
5:16 The first big question: why?
6:58 The second big question: what happened?
8:12 What’s the participant’s experience?
9:07 A tip for reading complex graphs.
10:23 The third big question: the explanation.
11:04 Some complexities when measuring learning
13:45 What’s the question again?
14:45 Last thoughts
If you want to be the first to know about the courses I'm releasing, sign up here: https://forms.gle/px7ZmXkvJW26uFWp8
Sign up to my newsletter, Avoiding Folly, here: benjaminkeep.com
REFERENCES
Clips come from the following pieces:
Example of a longitudinal study: Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363.
Example of qualitative data and mixed qualitative and quantitative data come from: Reading Abraham Lincoln: An Expert/Expert Study in the Interpretation of Historical Texts. (2003). 1–28.
Example of quantitative data comes from: Bjork, E. L., & Storm, B. C. (2011). Retrieval experience as a modifier of future encoding: Another test effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5), 1113–1124. doi.org/10.1037/a0023549
Example of abstract/laboratory experiments comes from: Bjork, R. A. (1975). Retrieval as a memory modifier: An interpretation of negative recency and related phenomena.
Example of an applied study (using social studies material in the classroom): Roediger, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in the classroom: Long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17(4), 382–395. doi.org/10.1037/a0026252
Examples of an experimental study, data visualization that describes the nature of the materials and one that describes the course of the experiment from: Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129–184.
Research studying the effects of delayed vs immediate feedback are legion. This provides good background reading: Fyfe, E. R., & Rittle-Johnson, B. (2016). Feedback both helps and hinders learning: The causal role of prior knowledge. Journal of Educational Psychology, 108(1), 82–97. doi.org/10.1037/edu0000053
The example of worked examples vs open problem solving comes from some of Sweller’s earlier experiments. Here’s a typical example. Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and instruction, 2(1), 59-89.
The statistics game and passage examples I drew from Arena, D. A., & Schwartz, D. L. (2013). Experience and Explanation: Using Videogames to Prepare Students for Formal Instruction in Statistics. Journal of Science Education and Technology, 23(4), 538–548. doi.org/10.1007/s10956-013-9483-3
The screenshot of a statistics passage is from one of the first pages of Regression and Other Stories by Gelman, Hill, and Vehtari, a great textbook on regression.
The design-based research example comes from Ketelhut, D. J., Nelson, B. C., Clarke, J., & Dede, C. (2010). A multi‐user virtual environment for building and assessing higher order inquiry skills in science. British Journal of Educational Technology, 41(1), 56-68. They created a virtual environment called “River City” to facilitate science inquiry experiences.
The “delayed measures” being different than immediate measures graph is from Taylor, K., & Rohrer, D. (2010). The effects of interleaved practice. Applied Cognitive Psychology, 24(6), 837–848. doi.org/10.1002/acp.1598
The “Stats Invaders” clip (the statistics game) comes courtesy of Can Zhang, here: youtube.com/watch?v=J7rx1g8g1Uk
Reading peer-reviewed social science research can be quite challenging. But it’s a skill that lets you go beyond media reports of the latest study. Here’s 15 minutes of advice on how to do read experimental studies specifically.
0:00 An intro to reading social science research
0:39 What do learning researchers study?
1:54 The basics of experimental designs
5:01 What’s the goal of reading research?
5:16 The first big question: why?
6:58 The second big question: what happened?
8:12 What’s the participant’s experience?
9:07 A tip for reading complex graphs.
10:23 The third big question: the explanation.
11:04 Some complexities when measuring learning
13:45 What’s the question again?
14:45 Last thoughts
If you want to be the first to know about the courses I'm releasing, sign up here: https://forms.gle/px7ZmXkvJW26uFWp8
Sign up to my newsletter, Avoiding Folly, here: benjaminkeep.com
REFERENCES
Clips come from the following pieces:
Example of a longitudinal study: Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363.
Example of qualitative data and mixed qualitative and quantitative data come from: Reading Abraham Lincoln: An Expert/Expert Study in the Interpretation of Historical Texts. (2003). 1–28.
Example of quantitative data comes from: Bjork, E. L., & Storm, B. C. (2011). Retrieval experience as a modifier of future encoding: Another test effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5), 1113–1124. doi.org/10.1037/a0023549
Example of abstract/laboratory experiments comes from: Bjork, R. A. (1975). Retrieval as a memory modifier: An interpretation of negative recency and related phenomena.
Example of an applied study (using social studies material in the classroom): Roediger, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in the classroom: Long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17(4), 382–395. doi.org/10.1037/a0026252
Examples of an experimental study, data visualization that describes the nature of the materials and one that describes the course of the experiment from: Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129–184.
Research studying the effects of delayed vs immediate feedback are legion. This provides good background reading: Fyfe, E. R., & Rittle-Johnson, B. (2016). Feedback both helps and hinders learning: The causal role of prior knowledge. Journal of Educational Psychology, 108(1), 82–97. doi.org/10.1037/edu0000053
The example of worked examples vs open problem solving comes from some of Sweller’s earlier experiments. Here’s a typical example. Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and instruction, 2(1), 59-89.
The statistics game and passage examples I drew from Arena, D. A., & Schwartz, D. L. (2013). Experience and Explanation: Using Videogames to Prepare Students for Formal Instruction in Statistics. Journal of Science Education and Technology, 23(4), 538–548. doi.org/10.1007/s10956-013-9483-3
The screenshot of a statistics passage is from one of the first pages of Regression and Other Stories by Gelman, Hill, and Vehtari, a great textbook on regression.
The design-based research example comes from Ketelhut, D. J., Nelson, B. C., Clarke, J., & Dede, C. (2010). A multi‐user virtual environment for building and assessing higher order inquiry skills in science. British Journal of Educational Technology, 41(1), 56-68. They created a virtual environment called “River City” to facilitate science inquiry experiences.
The “delayed measures” being different than immediate measures graph is from Taylor, K., & Rohrer, D. (2010). The effects of interleaved practice. Applied Cognitive Psychology, 24(6), 837–848. doi.org/10.1002/acp.1598
The “Stats Invaders” clip (the statistics game) comes courtesy of Can Zhang, here: youtube.com/watch?v=J7rx1g8g1Uk
