0
$\begingroup$

I am able to print nullity correlation matrix using Using pandas- df.isnull().corr() (this is how it is done is missingno). But what is the maths behind it ?How is nullity matrix calculated when missing data is present ?

$\endgroup$
2
$\begingroup$

You basically convert the columns into a boolean of is / is not null , and calculate the correlation:

import pandas as pd
import numpy as np
df = pd.DataFrame({'A':[0,np.NaN,np.NaN],'B':[np.NaN,0,np.NaN]})

A   B
0   0.0 NaN
1   NaN 0.0
2   NaN NaN

The first part does:

df.isnull()

    A   B
0   False   True
1   True    False
2   True    True

Then a correlation with df.isnull().corr()

      A      B
A   1.0     -0.5
B   -0.5    1.0

Which is the same as converting to zeros and ones, and doing the correlation:

df.isnull().astype(int)
    A   B
0   0   1
1   1   0
2   1   1
np.corrcoef(df.isnull().astype(int)['A'],df.isnull().astype(int)['B'])

array([[ 1. , -0.5],
       [-0.5,  1. ]])

Quick one on how to calculate correlation, which is covariance standardized by standard deviation. For example, covariance between column A and B it will be:

$\frac{1}{n-1}\sum_{i=1}^{n}(A_i-\bar{A})(B_i-\bar{B})$

In code it is:

A_val = df.isnull().astype(int)['A']
B_val = df.isnull().astype(int)['B']
n = len(A_val)
COV = np.sum((A_val-np.mean(A_val))*(B_val-np.mean(B_val)))/(n-1)
COV/np.std(A_val,ddof=1)/np.std(B_val,ddof=1)

-0.4999999999999999

You can see more about calculations here and here.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the beautiful explanation and the code.np.corrcoef , is it calculated by cov (a,b) / [ sd(a) * sd(b) ] ? @StupidWolf $\endgroup$ – Dev May 5 at 17:26
  • $\begingroup$ the covariance standardized by the standard deviation? You are asking about the math to calculate correlation? $\endgroup$ – StupidWolf May 5 at 17:29
  • $\begingroup$ yes.. should i write it in the answer for you? this is for the off diagonal.. the diagonals are always 1 $\endgroup$ – StupidWolf May 5 at 17:34
  • $\begingroup$ Yes Please , It would be more explanatory .Thanks In advance @StupidWolf $\endgroup$ – Dev May 5 at 17:38
  • $\begingroup$ Ok done. Hope it helps $\endgroup$ – StupidWolf May 5 at 18:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.