5
$\begingroup$

Currently I am working on a large data set with well over 200 variables (238 to be exact) and 290 observations for each variable (in theory). This data set is missing quite a lot of values, with variables ranging from 0-100% 'missingness'. I will eventually be performing logistical regression on this data, so of my 238 columns I will at most only be using ten or so.

However as almost all of my columns are missing some data, I am turning to multiple imputation to fill in the blanks (using the MICE package).

My question is; given that I have a large amount of variation in the missing data, at what percentage missing should I start to exclude variables from the mice() function?

Can mice function well with variables that are missing 50% of their values? What about 60%, 70%, 80%, 90%?

$\endgroup$

migrated from stackoverflow.com Apr 30 '15 at 23:24

This question came from our site for professional and enthusiast programmers.

4
$\begingroup$

In principle, MICE should be able to handle large amounts of missing data. Variables with lots of missing data points would be expected to end up with larger error terms than those with fewer missing data points, so your ability to detect significant relations to those variables would be limited accordingly. That's an advantage of having multiple imputations and analyzing results from all of the imputations.

More important than a "cutoff" for missing data is to consider carefully (1) the intended use of your model and (2) whether the "missing-at-random" assumptions needed for multiple imputation holds in your case.

In terms of (1) if you, say, intend to use the model for prediction but some variables are inherently hard to get, then there's no sense including them in the model. Also, you should use your knowledge of the subject matter to consider variables for inclusion. If you suspect that only 10 or so will be important based on such knowledge, maybe you should just use those 10.

In terms of (2), if the probability of missing data for a variable depends on the actual value of the variable, then multiple imputation is inappropriate.

$\endgroup$
1
$\begingroup$

This is not a coding question but if you want an answer here it is...

Missing data are very complicated. There is not a percentage value to accept of discard your variables. The variance of your variable is what is important to watch before imputation of data.

If you do not want to take some time to review all the statistic behind missing values, just take your variables with less missing value.

If you take the time read the MICE manual there is some basic information that will help you to impute correctly.

Missing data are not simple task, you have know what you do. Otherwise you will introduce bias!

$\endgroup$
  • $\begingroup$ I'm not sure that this really answers the question. $\endgroup$ – Hong Ooi Apr 30 '15 at 16:05
  • $\begingroup$ I think the advice to consider the variance is good. $\endgroup$ – Seanosapien Nov 18 '17 at 10:18
1
$\begingroup$

Mice can handle a large amount of missing data. Especially if there are a lot of columns with few missing data, one with 80% is no problem. You can also expect that in most of the times adding this variable leads to better imputation results than leaving it out. ( because more information / correlations available that help estimating the other variables)

But: The hard truth is, you will never know for sure, how good the imputation is anyway. Because the true values are well ... "missing"

If I'd have several imputation options to choose from, I'd take the one which leads to the best results for the prediction model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.