I am able to print nullity correlation matrix using Using pandas- df.isnull().corr() (this is how it is done is missingno). But what is the maths behind it ?How is nullity matrix calculated when missing data is present ?
1 Answer
You basically convert the columns into a boolean of is / is not null , and calculate the correlation:
import pandas as pd
import numpy as np
df = pd.DataFrame({'A':[0,np.NaN,np.NaN],'B':[np.NaN,0,np.NaN]})
A B
0 0.0 NaN
1 NaN 0.0
2 NaN NaN
The first part does:
df.isnull()
A B
0 False True
1 True False
2 True True
Then a correlation with df.isnull().corr()
A B
A 1.0 -0.5
B -0.5 1.0
Which is the same as converting to zeros and ones, and doing the correlation:
df.isnull().astype(int)
A B
0 0 1
1 1 0
2 1 1
np.corrcoef(df.isnull().astype(int)['A'],df.isnull().astype(int)['B'])
array([[ 1. , -0.5],
[-0.5, 1. ]])
Quick one on how to calculate correlation, which is covariance standardized by standard deviation. For example, covariance between column A and B it will be:
$\frac{1}{n-1}\sum_{i=1}^{n}(A_i-\bar{A})(B_i-\bar{B})$
In code it is:
A_val = df.isnull().astype(int)['A']
B_val = df.isnull().astype(int)['B']
n = len(A_val)
COV = np.sum((A_val-np.mean(A_val))*(B_val-np.mean(B_val)))/(n-1)
COV/np.std(A_val,ddof=1)/np.std(B_val,ddof=1)
-0.4999999999999999
-
$\begingroup$ Thanks for the beautiful explanation and the code.np.corrcoef , is it calculated by cov (a,b) / [ sd(a) * sd(b) ] ? @StupidWolf $\endgroup$– DevMay 5, 2020 at 17:26
-
$\begingroup$ the covariance standardized by the standard deviation? You are asking about the math to calculate correlation? $\endgroup$ May 5, 2020 at 17:29
-
$\begingroup$ yes.. should i write it in the answer for you? this is for the off diagonal.. the diagonals are always 1 $\endgroup$ May 5, 2020 at 17:34
-
$\begingroup$ Yes Please , It would be more explanatory .Thanks In advance @StupidWolf $\endgroup$– DevMay 5, 2020 at 17:38
-