# How to calculate nullity correlation matrix?

I am able to print nullity correlation matrix using Using pandas- df.isnull().corr() (this is how it is done is missingno). But what is the maths behind it ?How is nullity matrix calculated when missing data is present ?

You basically convert the columns into a boolean of is / is not null , and calculate the correlation:

import pandas as pd
import numpy as np
df = pd.DataFrame({'A':[0,np.NaN,np.NaN],'B':[np.NaN,0,np.NaN]})

A   B
0   0.0 NaN
1   NaN 0.0
2   NaN NaN


The first part does:

df.isnull()

A   B
0   False   True
1   True    False
2   True    True


Then a correlation with df.isnull().corr()

      A      B
A   1.0     -0.5
B   -0.5    1.0


Which is the same as converting to zeros and ones, and doing the correlation:

df.isnull().astype(int)
A   B
0   0   1
1   1   0
2   1   1
np.corrcoef(df.isnull().astype(int)['A'],df.isnull().astype(int)['B'])

array([[ 1. , -0.5],
[-0.5,  1. ]])


Quick one on how to calculate correlation, which is covariance standardized by standard deviation. For example, covariance between column A and B it will be:

$$\frac{1}{n-1}\sum_{i=1}^{n}(A_i-\bar{A})(B_i-\bar{B})$$

In code it is:

A_val = df.isnull().astype(int)['A']
B_val = df.isnull().astype(int)['B']
n = len(A_val)
COV = np.sum((A_val-np.mean(A_val))*(B_val-np.mean(B_val)))/(n-1)
COV/np.std(A_val,ddof=1)/np.std(B_val,ddof=1)

-0.4999999999999999


You can see more about calculations here and here.

• Thanks for the beautiful explanation and the code.np.corrcoef , is it calculated by cov (a,b) / [ sd(a) * sd(b) ] ? @StupidWolf
– Dev
May 5, 2020 at 17:26
• the covariance standardized by the standard deviation? You are asking about the math to calculate correlation? May 5, 2020 at 17:29
• yes.. should i write it in the answer for you? this is for the off diagonal.. the diagonals are always 1 May 5, 2020 at 17:34
• Yes Please , It would be more explanatory .Thanks In advance @StupidWolf
– Dev
May 5, 2020 at 17:38
• Ok done. Hope it helps May 5, 2020 at 18:03