Uploaded April 2019 | Updated September 2026, 1 hour ago
This one's inspired by a common confidence interval misinterpretation.
(This is a bit of a different video for me, and if you're just looking for help with specific topics in a statistics course, you may not find it helpful. But there's some good stuff in here.)
Here I address what might seem at first like bit of a strange or uninformative question: In repeated sampling from a normally distributed population, on average what proportion of sample means would a randomly selected 95% CI for mu capture? I work through the calculations, then I discuss how this notion relates to bad confidence interval interpretations and reproducibility* studies.
This was inspired by a bad confidence interval interpretation that I heard a number of years ago (and have heard variants of ever since), where, when interpreting a 95% confidence interval for the population mean, the individual stated:
``If you repeat the same study a million times, then the mean of each one of those samples should fall in the interval 95% of the time.'' [Edited slightly to improve the readability.]
This is a poor interpretation of the interval, and simply untrue. It's just not the case. So, then, what is the probability a randomly selected 95% confidence interval for mu captures the mean of another sample of the same size from the same population?
This has applications in reproducibility* studies, and I briefly discuss that after working through the calculations. My discussion is not intended to be a complete discussion of issues in reproducibility*, just a brief discussion of how the question I answer relates.
*In this video I use "reproducible" and "replicable" interchangeably. I know there has been much discussion in some circles of differences between those terms. Apologies if you find my casual use of these terms problematic or misleading.
Reference for the paper I bring up:
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. \textit{Science}, 349(6251), 1--8.
Reference for a different discussion about how there is extra variability when comparing two statistics, and why the results found in the reference above might not be as bad as they appear at first blush:
Patil P., Peng R. D., Leek J. T. (2016). What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 539--544. 10.1177/1745691616646366
This one's inspired by a common confidence interval misinterpretation.
(This is a bit of a different video for me, and if you're just looking for help with specific topics in a statistics course, you may not find it helpful. But there's some good stuff in here.)
Here I address what might seem at first like bit of a strange or uninformative question: In repeated sampling from a normally distributed population, on average what proportion of sample means would a randomly selected 95% CI for mu capture? I work through the calculations, then I discuss how this notion relates to bad confidence interval interpretations and reproducibility* studies.
This was inspired by a bad confidence interval interpretation that I heard a number of years ago (and have heard variants of ever since), where, when interpreting a 95% confidence interval for the population mean, the individual stated:
``If you repeat the same study a million times, then the mean of each one of those samples should fall in the interval 95% of the time.'' [Edited slightly to improve the readability.]
This is a poor interpretation of the interval, and simply untrue. It's just not the case. So, then, what is the probability a randomly selected 95% confidence interval for mu captures the mean of another sample of the same size from the same population?
This has applications in reproducibility* studies, and I briefly discuss that after working through the calculations. My discussion is not intended to be a complete discussion of issues in reproducibility*, just a brief discussion of how the question I answer relates.
*In this video I use "reproducible" and "replicable" interchangeably. I know there has been much discussion in some circles of differences between those terms. Apologies if you find my casual use of these terms problematic or misleading.
Reference for the paper I bring up:
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. \textit{Science}, 349(6251), 1--8.
Reference for a different discussion about how there is extra variability when comparing two statistics, and why the results found in the reference above might not be as bad as they appear at first blush:
Patil P., Peng R. D., Leek J. T. (2016). What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 539--544. 10.1177/1745691616646366
