Interpreting a nullity correlation matrix

Question

I learning data science using Dataquest. In one of the missions, I encountered a nullity correlation plot like this:

I am already familiar with the concept of correlation. But I have never heard of "nullity correlation".

My question is what is nullity correlation and how can I interpret this plot?

Answer 1

This is a technique used to understand patterns of missing values in a dataset.

Given two variables $X$ and $Y$ (which are in effect equal-length arrays of values, which are either references to computer objects -- of any type -- or a special code to representing a missing value), we can quantify their patterns of missing values using some kind of "correlation coefficient."

The most suitable correlation coefficients to use would be those developed for binary data, because we are really just thinking of the data as being "non-missing" or "missing," without regard to the actual values of the non-missing entries. Examples include Pearson's $\phi$, Matthews' MCC, Cramér's $\phi_C$, and Tetrachoric correlation.

However, a little Web searching suggests instead that people disregard any such subtleties and are just using the Pearson correlation coefficient. See, for instance, the answer at https://datascience.stackexchange.com/questions/61944/how-to-print-nullity-correlation-matrix, which correctly relates what's in the source code on GitHub. This suggests the technique is intended for use on medium to large datasets, where the distinctions among the different types of correlation are relatively unimportant.

The Pearson coefficient $\rho$ has a simple formula when data are binary. Since it doesn't matter how the missingness is coded, suppose (with no loss in generality) that all missing values in both vectors are replaced by the number $1$ and every other (non-missing) value is replaced by the number $0:$ this is the indicator of missingness, $\mathcal{I}.$

Pearson's $\rho$ is defined in terms of covariances. One formula for the covariance of two arrays of numbers $a$ and $b$ (necessarily of the same length $n,$ which I will index from $1$ through $n$) is

$$\operatorname{Cov}(a,b) = \frac{1}{n}\sum_{i=1}^n a_i b_i - \left(\frac{1}{n}\sum_{i=1}^n a_i\right)\left(\frac{1}{n}\sum_{j=1}^n b_j\right).\tag{*}$$

(There are other formulas that differ from this one by a constant positive multiple $C(n).$ As we will see, these all give the same value of $\rho.$)

The definition of Pearsons' coefficient is

$$\rho(a,b) = \frac{\operatorname{Cov}(a,b)}{\sqrt{\operatorname{Cov}(a,a)\operatorname{Cov}(b,b)}}$$

provided the denominator is nonzero. (This formula makes it evident that multiplying all covariances by $C(n)$ multiplies $\rho$ by $C(n)/\sqrt{C(n)^2} = 1,$ demonstrating the value of $C(n)$ doesn't matter.)

To apply this to patterns of missingness in $X$ and $Y$, notice that the calculations in $(*)$ come down to counting four things:

1. $n_{00}$ is the number of places at which both $X$ and $Y$ are non-missing.
2. $n_{01}$ is the number of places at which $X$ has a non-missing value but $Y$ has a missing value.
3. $n_{10}$ is the number of places at which $X$ has a missing value but $Y$ has a non-missing value.
4. $n_{11}$ is the number of places at which both $X$ and $Y$ are missing.

$$\begin{array}{l|cc|c} &Y ~ \text{not missing} & Y ~ \text{missing} & \text{Total} \\ \hline X ~ \text{not missing} & n_{00} & n_{01} & n_{0\cdot}\\ X ~ \text{missing} & n_{10} & n_{11} & n_{1\cdot}\\ \hline\text{Total} & n_{\cdot 0} & n_{\cdot 1} & n \end{array}$$

In particular, the number of missing values in $X$ is $n_{1\cdot} = n_{10}+n_{11}$ and the number of missing values in $Y$ is $n_{\cdot1} = n_{01}+n_{11}.$

In terms of these counts, $(*)$ applied to $a=\mathcal{I}(X)$ and $b=\mathcal{I}(Y)$ simplifies to

$$\rho(\mathcal{I}(X), \mathcal{I}(Y)) = \frac{n n_{11} - n_{1\cdot}n_{\cdot1}}{\sqrt{(n n_{1\cdot}-n_{1\cdot}^2)(n n_{\cdot1}-n_{\cdot1}^2)}}.$$

This formula is useful in revealing that

• $\rho$ increases with increasing deviation between $n_{11}/n$ (the proportion of entries in which both arrays have missing values) and $(n_{1\cdot}/n)\,(n_{\cdot 1}/n)$ (the proportion of such entries one would predict based on the separate proportions in the arrays).

• Small values in the denominator can make $|\rho|$ relatively large. That happens when in both arrays there are either few missing values or few non-missing values.

This alerts us to a subtlety in interpreting $\rho:$ we eventually will want to pay attention to the proportions of missing values in the variables. (It is unfortunate that the heat map in the question does not indicate these proportions -- there is room for improvement in this regard.)

The usual properties of Pearson's coefficient apply: in particular, the values of $\rho$ lie between $-1$ and $1.$

• When $\rho=1$ it means $X$ and $Y$ have identical missingness patterns (that is, there are some missing values and some non-missing values in both arrays and the missing values occur at exactly the same locations in both arrays).

• When $\rho=-1$ it means $X$ and $Y$ have opposite missingness patterns (that is, there are some missing values and some non-missing values in both arrays and when one array's value is missing the other is non-missing and vice versa).

• A value of $\rho$ near $0$ means the missingness patterns of $X$ and $Y$ appear to be unrelated to each other.

Generalizing from this, but (necessarily) becoming more qualitative, a value of $\rho$ close to $1$ means the missingness patterns in $X$ and $Y$ are "close" to being the same; a value close to $-1$ means the missingness patterns are "close" to being opposite; and a value close to $0$ means the missingness patterns in $X$ and $Y$ are "nearly" unrelated.

The "nullity correlation matrix" is a square array of all these nullity correlation coefficients. Its rows and columns correspond to variables of interest for which there are (a) some missing values and (b) some non-missing values.

The image in the question is a "heat map" displaying a medium-size nullity correlation matrix in a way that helps quickly identify pairs of variables with similar missingness patterns (dark blue), nearly opposite missingness patterns (dark red), and nearly unrelated missingness patterns (white).

A glance at this matrix, then, is helpful in assessing hypotheses about whether pairs of variables might be lacking data for similar, opposite, or unrelated reasons.

Answer 2

Nullity correlation ranges from -1 to 1. Zero indicates the variables do not have any correlation. -1 indicates a strong negative correlation where +1 indicates a strong positive correlation (think of it as the direction of slope)

Answer 3

Association of null value in columns is how much it correlates with other column

For nullinity matrix correlation calculation Refer, How to calculate nullity correlation matrix?

Ex: In house price calculation, we will gather swimming pool and gardening data only if it is individual home. If it is appartment, we will gather the data. In this case it is filled as nan instead of zero in data collection process.

So if we have nan in swimming pool if it is appartment type. It is a missing value intentionally. i.e., known missing value with purpose.