I have this huge data set with like 2500 variables and like 142 observations.

I want to run a correlation between Variable X and the rest of the variables. But for many columns, there are entries missing.

I tried to do this in R using "pairwise-complete" argument (use=pairwise.complete.obs) and it outputted a bunch of correlations. But then someone on StackOverflow posted a link to this article http://bwlewis.github.io/covar/missing.html and it makes the "pairwise-complete" method in R look unusable.

My Question: How do I know when it is appropriate to use "pairwise-complete" option?

My use = complete.obs returned no complete element pairs, so if you could explain what that means too, that would be great.


The issue with correlations on pairwise complete observations

In the case you describe, the main issue is interpretation. Because you're using pairwise complete observations, you are actually analyzing slightly different datasets for each of the correlations, depending on which observations are missing.

Consider the following example:

a <- c(NA,NA,NA, 5, 6, 3, 7, 8, 3)
b <- c(2, 8, 3, NA,NA,NA, 6, 9, 5)
c <- c(2, 9, 6, 3, 2, 3, NA,NA,NA) 

Three variables in the dataset, a, b, and c, each has some missing values. If you calculate correlations on pairs of variables here, you'll only be able to use cases that don't have missing values for both of the variables in question. In this case, that means you'll be analyzing just the last 3 cases for the correlation between a and b, just the first three cases for the correlation between b and c, etc.

The fact that you're analyzing completely different cases when you calculate each correlation means that the resulting pattern of correlations can look nonsensical. See:

> cor(a,b, use = "pairwise.complete.obs")
[1] 0.8170572
> cor(b,c, use = "pairwise.complete.obs")
[1] 0.9005714
> cor(a,c, use = "pairwise.complete.obs")
[1] -0.7559289

This looks like a logical contradiction --- a and b are strongly positively correlated, and b and c are also strongly positively correlated, so you would expect a and c to be positively correlated as well, but there's actually a strong association in the opposite direction. You can see why a lot of analysts don't like that.

Edit to include useful clarification from whuber:

Note that part of the argument depends on what "strong" correlation might mean. It is quite possible for a and b as well as b and c to be "strongly positively correlated" while there exists a "strong association in the opposite direction" between a and c, but not quite as extreme as in this example. The crux of the matter is that the estimated correlation (or covariance) matrix might not be positive-definite: that's how one should quantify "strong".

The issue with the type of missingness

You may be thinking to yourself, "Well, isn't it okay to just assume that the subset of cases I have available for each correlation follow more or less the same pattern I would get if I had complete data?" And yes, that's true --- there's nothing fundamentally wrong with calculating a correlation on a subset of your data (although you lose precision and power, of course, because of the smaller sample size), as long as the available data are a random sample of all of the data that would have been there if you didn't have any missingness.

When the missingness is purely random, that's called MCAR (missing completely at random). In that case, analyzing the subset of the data that doesn't have missingness won't systematically bias your results, and it would be unlikely (but not impossible) to get the kind of nutsy correlation pattern I showed in the example above.

When your missingness is systematic in some way (often abbreviated MAR or NI, delineating two different kinds of systematic missingness) then you have much more serious issues, both in terms of potentially introducing bias in your calculations and in terms of your ability to generalize your results to the population of interest (because the sample you're analyzing is not a random sample from the population, even if your full dataset would have been).

There are a lot of great resources available to learn about missing data and how to deal with it, but my recommendation is Rubin: a classic, and a more recent article

| cite | improve this answer | |
  • 2
    $\begingroup$ +1. Note that part of your argument depends on what "strong" correlation might mean. It is quite possible for $a$ and $b$ as well as $b$ and $c$ to be "strongly positively correlated" while there exists a "strong association in the opposite direction" between $a$ and $c$. However, it's not possible for all three correlation coefficients to be quite as extreme as in your example, so you're ok there. The crux of the matter is that the estimated correlation (or covariance) matrix might not be positive-definite: that's how one should quantify "strong". $\endgroup$ – whuber Feb 20 '17 at 17:20
  • 1
    $\begingroup$ @whuber Thanks, that's an important point. I've updated that section of the answer to include that clarification. $\endgroup$ – Rose Hartman Feb 20 '17 at 18:11

A huge concern is whether data is missing in some systematic way which would corrupt your analysis. Your data may be missing not at random.

This was brought up in previous answers, but I thought I'd contribute an example.

Finance example: missing returns may be poor returns

  • Unlike mutual funds, private equity funds (and other private funds) are not required by law to report their returns to some central database.
  • Hence a major concern is that reporting is endogenous, more specifically, that some firms won't report bad returns.
  • If so, your average of reported fund returns $\frac{1}{n} \sum_i R_i$ will overestimate the true mean because low $R_i$ tend to be missing.

All is not necessarily lost in these situations (there are things you can do), but naively running a regression (or computing correlations) on the non-missing data may lead to seriously biased, inconsistent estimates of the true parameters in the population.

| cite | improve this answer | |

Pairwise correlation is appropriate if your missing data is Missing Complete At Random (MCAR). Paul Allison's Missing Data book is a good place to start for why.

You can test this using Little's (1988) MCAR Test, which is in the BaylorEdPsych package.

| cite | improve this answer | |
  • 1
    $\begingroup$ There's still cause for concern: even with MCAR data, the correlation matrix estimated via pairwise correlation can fail to be positive-definite. $\endgroup$ – whuber Feb 20 '17 at 17:21
  • $\begingroup$ Sure, but the question asks about correlation, it does not make any mention of the use of the resulting correlation matrix as an input to some other algorithm. And, given the sample size, MCAR is pretty unlikely anyway. $\endgroup$ – Tim Feb 20 '17 at 21:03
  • 1
    $\begingroup$ If the matrix is not positive-definite, it's an invalid estimate. At the least we have to be concerned about that inconsistency. I'm afraid I don't see how the likelihood of MCAR (which is a mechanism of missingness) could be related to the sample size. $\endgroup$ – whuber Feb 20 '17 at 23:02
  • $\begingroup$ The asker is interested in a single row of the correlation matrix. Have you got a proof that shows the correlations a row are all invalid if the matrix is not positive-definite? I would love to see a proof of this and gain some wisdom. MCAR is, in general, pretty unlikely with real world data. With a large sample size, the power of Little's test increases, so there is a good chance of rejection of the null hypothesis of MCAR. Don't get me wrong here: I would never use a partial-data correlation matrix as an input into a multivariate method, but this is not what the question asks about. $\endgroup$ – Tim Feb 21 '17 at 3:50
  • 1
    $\begingroup$ Let me clarify: I did not claim the correlations are "all invalid." I claimed that the collection of correlation estimates (that is, the matrix) can be invalid. That is indisputable (requiring no proof), because all one need do is exhibit one instance of an invalid estimate, which @RoseHartman has already done in this thread. I won't dispute your claim that MCAR might be unlikely--provided it is understood in a personal sense: in your experience, with the kinds of data you are familiar with, MCAR is rare. I don't see how you can justify any broader interpretation of that claim. $\endgroup$ – whuber Feb 21 '17 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.