How to calculate nullity correlation matrix?

Question

I am able to print nullity correlation matrix using Using pandas- df.isnull().corr() (this is how it is done is missingno). But what is the maths behind it ?How is nullity matrix calculated when missing data is present ?

Answer 1

You basically convert the columns into a boolean of is / is not null , and calculate the correlation:

import pandas as pd
import numpy as np
df = pd.DataFrame({'A':[0,np.NaN,np.NaN],'B':[np.NaN,0,np.NaN]})

A   B
0   0.0 NaN
1   NaN 0.0
2   NaN NaN

The first part does:

df.isnull()

    A   B
0   False   True
1   True    False
2   True    True

Then a correlation with df.isnull().corr()

      A      B
A   1.0     -0.5
B   -0.5    1.0

Which is the same as converting to zeros and ones, and doing the correlation:

df.isnull().astype(int)
    A   B
0   0   1
1   1   0
2   1   1
np.corrcoef(df.isnull().astype(int)['A'],df.isnull().astype(int)['B'])

array([[ 1. , -0.5],
       [-0.5,  1. ]])

Quick one on how to calculate correlation, which is covariance standardized by standard deviation. For example, covariance between column A and B it will be:

$\frac{1}{n-1}\sum_{i=1}^{n}(A_i-\bar{A})(B_i-\bar{B})$

In code it is:

A_val = df.isnull().astype(int)['A']
B_val = df.isnull().astype(int)['B']
n = len(A_val)
COV = np.sum((A_val-np.mean(A_val))*(B_val-np.mean(B_val)))/(n-1)
COV/np.std(A_val,ddof=1)/np.std(B_val,ddof=1)

-0.4999999999999999

You can see more about calculations here and here.