3
$\begingroup$

I have a dataset with 500K individuals and 200 variables. The problem we have is that most of the variables have a high percentage of missing values (~20 variables are 100% populated but other variables have less than 20% inputted values).

Before looking at the dataset, I was thinking about using the EM algorithm to populate the missing fields (at least for some of the most important variables). But looking at the high percentage of missing values, I am now wondering if it's still possible ... Do you have a statistical method to estimate what would be the minimum % of missing values to have to have significant results? If EM is not possible, do you know another methods to deal with this high proportion of missings?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Whether or not EM operates does not depend on the percentage of missing values but on the underlying model. What is your statistical model? Obviously, the more missing values, the more imprecise the inference. $\endgroup$
    – Xi'an
    Commented Oct 19, 2015 at 8:31

1 Answer 1

Reset to default
0
$\begingroup$

I'll first note that statistical significance really isn't what we should be caring about here with respect to missingness. Certainly missing values influence $p$ values, but we really should be concerned with whether or not the model is correctly specified, and that this missingness is properly accounted for.

One doesn't need to meet a magic number with missingness to address it, and it is often the case that if we have a large percentage of missingness, then imputation is critical to obtaining valid estimates. Normally, we would want to identify what kind of missingness mechanism is present first (MNAR, MAR, MCAR) before we do anything specific about it (MAR for example posits that there is some kind of relationship between a variable and missingness, which can be identified with some simple plotting or fitting to models which identify the culprits). These mechanisms are addressed in Little et al. (2014) and other places. Note that there are some toggles on the switches to consider, such as how many imputations we need if there is substantial missingness (see Graham et al., 2007 for simulations on multiple imputation), but usually the takeaway is that more imputations are better.

Common approaches are full information maximum likelihood, multiple imputation, and random forest imputation (Enders, 2001; Tang & Ishwaran, 2017; van Buuren & Groothuis-Oudshoorn, 2011). I know squat about the EM algorithm, so others may comment on whether or not it performs better than the more common methods I listed.

References

  • Enders, C. K. (2001). The performance of the full information maximum likelihood estimator in multiple regression models with missing data. Educational and Psychological Measurement, 61(5), 713–740.
  • Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8(3), 206–213. https://doi.org/10.1007/s11121-007-0070-9
  • Little, T. D., Jorgensen, T. D., Lang, K. M., & Moore, E. W. G. (2014). On the joys of missing data. Journal of Pediatric Psychology, 39(2), 151–162. https://doi.org/10.1093/jpepsy/jst048
  • Tang, F., & Ishwaran, H. (2017). Random forest missing data algorithms. Statistical Analysis and Data Mining: The ASA Data Science Journal, 10(6), 363–377. https://doi.org/10.1002/sam.11348
  • van Buuren, S., & Groothuis-Oudshoorn, K. (2011). MICE: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3). https://doi.org/10.18637/jss.v045.i03
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.