Today I read an interesting statement, which I am not sure about its correctness.

Assume we are looking at a data set with one column $H$ that is numeric. It has a bunch of missing values. Now let's say I just replace the missing value in column $H$ by the mean of this column (calculated by excluding the missing value). This method is good in the sense that it won't change the mean of column H. But it also has some problems. I was told that this will lessen the correlation with other numeric columns.

This is something I don't understand. I think the extreme case. Assume that column H only has 2 non-missing value, then I use the average of these 2 numbers to replace all the missing values in column H, then this will make the column H almost like a constant, which has variance close to 0. Then the correlation between this almost constant variable and other numeric column will be infinity? or undefined? or zero?


Consider the formula for computing the sample correlation between two variables, $x$ and $y$:

$$r_{x,y}=\frac{\sum_n^N(x_n-\bar x)(y_n-\bar y)}{(N-1)s_xs_y},$$

where $s_x$ and $s_y$ are non-zero corrected sample standard deviations. Now, if $x$ is the imputed variable, in all the imputed samples the term $(x_n-\bar x)$ will evaluate to 0, reducing the value of the numerator and thus of the correlation.

In your specific example, the the correlation between $H$ and some other numeric column would depend on the correlation between the values in non-imputed rows. It will always be in the range $[-1;1]$, unless one of the columns has zero sample variance, in which case it will be undefined.

| cite | improve this answer | |
  • $\begingroup$ If neither column x and y don't have missing data, then $r_{x,y}$ is the sample correlation. Now let's say in x column, we missed a couple of data and we fill them with the sample mean computed from the remaining non-missing data in column x (which is almost equal to the original sample mean). Now, for those filled row, the term $x_n-\bar{x}$ is 0. So the numerator shrinks. But for the denominator, when we compute $s_x$, it also gets smaller, right? Because $(x_n-\bar{x})^2$ for those filled rows are zero as well. So both numerator and denominator gets smaller. How can we tell about quotient? $\endgroup$ – KevinKim Feb 19 '18 at 15:21
  • $\begingroup$ The important part is that the zero terms in the nominator ($x_n-\bar x$) also cancel out the $y$-part of the nominator (sum of products). In the denominator, those variances are computed separately (product of sums), so the variance in $y$ is not cancelled out. $\endgroup$ – Jan Kukacka Feb 19 '18 at 15:28
  • $\begingroup$ I agree with you. But if the row with missing data of x has a small value of $y_n-\bar{y}$ and that row originally has a big value of $x_n-\bar{x}$, then the shrinkage in the numerator is kind of small but the shrinkage of the denominator is relatively large, as it is squared, right? Sorry for my being picky. I just want to be a little bit more precise. I agree with you in general. I just want to make sure that mathematically, there are some extreme corner case such that something else could happen $\endgroup$ – KevinKim Feb 19 '18 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.