1
$\begingroup$

I am able to print nullity correlation matrix using Using pandas- df.isnull().corr() (this is how it is done is missingno). But what is the maths behind it ?How is nullity matrix calculated when missing data is present ?

$\endgroup$

1 Answer 1

4
$\begingroup$

You basically convert the columns into a boolean of is / is not null , and calculate the correlation:

import pandas as pd
import numpy as np
df = pd.DataFrame({'A':[0,np.NaN,np.NaN],'B':[np.NaN,0,np.NaN]})

A   B
0   0.0 NaN
1   NaN 0.0
2   NaN NaN

The first part does:

df.isnull()

    A   B
0   False   True
1   True    False
2   True    True

Then a correlation with df.isnull().corr()

      A      B
A   1.0     -0.5
B   -0.5    1.0

Which is the same as converting to zeros and ones, and doing the correlation:

df.isnull().astype(int)
    A   B
0   0   1
1   1   0
2   1   1
np.corrcoef(df.isnull().astype(int)['A'],df.isnull().astype(int)['B'])

array([[ 1. , -0.5],
       [-0.5,  1. ]])

Quick one on how to calculate correlation, which is covariance standardized by standard deviation. For example, covariance between column A and B it will be:

$\frac{1}{n-1}\sum_{i=1}^{n}(A_i-\bar{A})(B_i-\bar{B})$

In code it is:

A_val = df.isnull().astype(int)['A']
B_val = df.isnull().astype(int)['B']
n = len(A_val)
COV = np.sum((A_val-np.mean(A_val))*(B_val-np.mean(B_val)))/(n-1)
COV/np.std(A_val,ddof=1)/np.std(B_val,ddof=1)

-0.4999999999999999

You can see more about calculations here and here.

$\endgroup$
5
  • $\begingroup$ Thanks for the beautiful explanation and the code.np.corrcoef , is it calculated by cov (a,b) / [ sd(a) * sd(b) ] ? @StupidWolf $\endgroup$
    – Dev
    May 5, 2020 at 17:26
  • $\begingroup$ the covariance standardized by the standard deviation? You are asking about the math to calculate correlation? $\endgroup$
    – StupidWolf
    May 5, 2020 at 17:29
  • $\begingroup$ yes.. should i write it in the answer for you? this is for the off diagonal.. the diagonals are always 1 $\endgroup$
    – StupidWolf
    May 5, 2020 at 17:34
  • $\begingroup$ Yes Please , It would be more explanatory .Thanks In advance @StupidWolf $\endgroup$
    – Dev
    May 5, 2020 at 17:38
  • $\begingroup$ Ok done. Hope it helps $\endgroup$
    – StupidWolf
    May 5, 2020 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.