Uploaded July 2022 | Updated September 2026, 2 weeks ago
Let’s talk about #Correlation test in Data Science with Python, Pandas and Scipy libraries. This video provides the overview between pandas.dataframe.corr() method and Scipy for calculating a coefficient of correlation for you features stored in Pandas DataFrame. The advantage comes from Scipy is that you will be able to evaluate the importance of the calculated correlation and decide which one is meaningful, and which ones are not for you business problem.
First of all, let’s be familiar with the official pd.DataFrame.corr() method documentation here: pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html
As you can see, this way by using Pandas provides limited flexibility to use correlation for you Data Science project. There you can change the method of calculation (pearson, kendall, and spearman) and min_periods parameters only. So, it has minimal configuration.
From my personal experience, I suggest to try another way to play with correlation. You can test them with #pvalue which can be automatically calculated in you #Pandas Dataframe.
A mandatory warning that must be mentioned when talking about correlation is “Correlation does not imply causation”. You can read more about this warning in this Medium.com article: sundaskhalid.medium.com/correlation-vs-causation-in-data-science-66b6cfa702f0
You would often use correlation during exploratory data analysis (EDA) both for supervised and unsupervised ML problems to solve. The most straighforward way to calculate correlation is to use built-in Pandas method for that: pandas.dataframe.corr(). You can see the example in the video and in this Github code gist: gist.githubusercontent.com/KarunThankachan/f7b297630436042ca3e4f4c45f799b0a/raw/4f0bf26d10cc9d7ebafbc33abea09a69aaa30591/medium_corr.py
THE CORE OF THIS VIDEO:
When we talk about the correlation between variables, what we ideally want to measure is the correlation between variables in the entire population. However most data scientists work with a sample of data. So if we obtain a different sample it’s possible we could have different correlation scores. As such we need to assess the significance of the correlation values we calculated, which depends on the sample size.
By using Pandas approach, you will never get a p-values for your correlation, so you will be not able to test them. I suggest use Scipy library for this reason. Scipy’s stats library offers all three versions of the correlation test offered in pandas.DataFrame.corr():
- Pearson (docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html)
- Spearman (docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html)
- KendalTau (docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html)
By using Scipy approach, you can pass columns of dataframe you want to compare. The Github snippet is here: gist.githubusercontent.com/KarunThankachan/6782cfedf3991248ca80c93332ca3d3f/raw/4fe0ec63a9d2598941b5e5c39ddbec9a5e3c17f5/medium_corr_v2.py
From the output of the code linked above, we can see the p-values and hence know how significant the correlations are.
Conclusion:
For correlation tests on data samples (which is often what you will be working on) always calculate the p-values as well. As such, when working with sample data go for the scipy.stats over pandas.DataFrame.corr()
Read more about:
- Pearson and Spearman correlation coefficients (clearly explained): towardsdatascience.com/clearly-explained-pearson-v-s-spearman-correlation-coefficient-ada2f473b8
- Kendall Rank Correlation (explained): towardsdatascience.com/kendall-rank-correlation-explained-dee01d99c535
- Interpreting correlations: towardsdatascience.com/eveything-you-need-to-know-about-interpreting-correlations-2c485841c0b8
Let’s talk about #Correlation test in Data Science with Python, Pandas and Scipy libraries. This video provides the overview between pandas.dataframe.corr() method and Scipy for calculating a coefficient of correlation for you features stored in Pandas DataFrame. The advantage comes from Scipy is that you will be able to evaluate the importance of the calculated correlation and decide which one is meaningful, and which ones are not for you business problem.
First of all, let’s be familiar with the official pd.DataFrame.corr() method documentation here: pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html
As you can see, this way by using Pandas provides limited flexibility to use correlation for you Data Science project. There you can change the method of calculation (pearson, kendall, and spearman) and min_periods parameters only. So, it has minimal configuration.
From my personal experience, I suggest to try another way to play with correlation. You can test them with #pvalue which can be automatically calculated in you #Pandas Dataframe.
A mandatory warning that must be mentioned when talking about correlation is “Correlation does not imply causation”. You can read more about this warning in this Medium.com article: sundaskhalid.medium.com/correlation-vs-causation-in-data-science-66b6cfa702f0
You would often use correlation during exploratory data analysis (EDA) both for supervised and unsupervised ML problems to solve. The most straighforward way to calculate correlation is to use built-in Pandas method for that: pandas.dataframe.corr(). You can see the example in the video and in this Github code gist: gist.githubusercontent.com/KarunThankachan/f7b297630436042ca3e4f4c45f799b0a/raw/4f0bf26d10cc9d7ebafbc33abea09a69aaa30591/medium_corr.py
THE CORE OF THIS VIDEO:
When we talk about the correlation between variables, what we ideally want to measure is the correlation between variables in the entire population. However most data scientists work with a sample of data. So if we obtain a different sample it’s possible we could have different correlation scores. As such we need to assess the significance of the correlation values we calculated, which depends on the sample size.
By using Pandas approach, you will never get a p-values for your correlation, so you will be not able to test them. I suggest use Scipy library for this reason. Scipy’s stats library offers all three versions of the correlation test offered in pandas.DataFrame.corr():
- Pearson (docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html)
- Spearman (docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html)
- KendalTau (docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html)
By using Scipy approach, you can pass columns of dataframe you want to compare. The Github snippet is here: gist.githubusercontent.com/KarunThankachan/6782cfedf3991248ca80c93332ca3d3f/raw/4fe0ec63a9d2598941b5e5c39ddbec9a5e3c17f5/medium_corr_v2.py
From the output of the code linked above, we can see the p-values and hence know how significant the correlations are.
Conclusion:
For correlation tests on data samples (which is often what you will be working on) always calculate the p-values as well. As such, when working with sample data go for the scipy.stats over pandas.DataFrame.corr()
Read more about:
- Pearson and Spearman correlation coefficients (clearly explained): towardsdatascience.com/clearly-explained-pearson-v-s-spearman-correlation-coefficient-ada2f473b8
- Kendall Rank Correlation (explained): towardsdatascience.com/kendall-rank-correlation-explained-dee01d99c535
- Interpreting correlations: towardsdatascience.com/eveything-you-need-to-know-about-interpreting-correlations-2c485841c0b8
