1
$\begingroup$

I am able to print nullity correlation matrix using Using pandas- df.isnull().corr() (this is how it is done is missingno). But what is the maths behind it ?How is nullity matrix calculated when missing data is present ?

$\endgroup$
3
$\begingroup$

You basically convert the columns into a boolean of is / is not null , and calculate the correlation:

import pandas as pd
import numpy as np
df = pd.DataFrame({'A':[0,np.NaN,np.NaN],'B':[np.NaN,0,np.NaN]})

A   B
0   0.0 NaN
1   NaN 0.0
2   NaN NaN

The first part does:

df.isnull()

    A   B
0   False   True
1   True    False
2   True    True

Then a correlation with df.isnull().corr()

      A      B
A   1.0     -0.5
B   -0.5    1.0

Which is the same as converting to zeros and ones, and doing the correlation:

df.isnull().astype(int)
    A   B
0   0   1
1   1   0
2   1   1
np.corrcoef(df.isnull().astype(int)['A'],df.isnull().astype(int)['B'])

array([[ 1. , -0.5],
       [-0.5,  1. ]])

Quick one on how to calculate correlation, which is covariance standardized by standard deviation. For example, covariance between column A and B it will be:

$\frac{1}{n-1}\sum_{i=1}^{n}(A_i-\bar{A})(B_i-\bar{B})$

In code it is:

A_val = df.isnull().astype(int)['A']
B_val = df.isnull().astype(int)['B']
n = len(A_val)
COV = np.sum((A_val-np.mean(A_val))*(B_val-np.mean(B_val)))/(n-1)
COV/np.std(A_val,ddof=1)/np.std(B_val,ddof=1)

-0.4999999999999999

You can see more about calculations here and here.

$\endgroup$
5
  • $\begingroup$ Thanks for the beautiful explanation and the code.np.corrcoef , is it calculated by cov (a,b) / [ sd(a) * sd(b) ] ? @StupidWolf $\endgroup$ – Dev May 5 '20 at 17:26
  • $\begingroup$ the covariance standardized by the standard deviation? You are asking about the math to calculate correlation? $\endgroup$ – StupidWolf May 5 '20 at 17:29
  • $\begingroup$ yes.. should i write it in the answer for you? this is for the off diagonal.. the diagonals are always 1 $\endgroup$ – StupidWolf May 5 '20 at 17:34
  • $\begingroup$ Yes Please , It would be more explanatory .Thanks In advance @StupidWolf $\endgroup$ – Dev May 5 '20 at 17:38
  • $\begingroup$ Ok done. Hope it helps $\endgroup$ – StupidWolf May 5 '20 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.