I am planning to carry out multiple imputation on a large data set with around 4,000,000 observations and 32 variables and 10 interactions. Of these variables six have missing data which I need to impute.

I plan to use MICE package from R and the fully conditionally specified method, outlined here: https://www.jstatsoft.org/article/view/v045i03

I have tried to run a test imputation using a subset of 10,000 observations, for 1 iteration and 5 imputation data sets and it is taking very long to run, so I am worried how long it will take when I try it with the full data set. I have spent quite a lot of time properly understanding how this package works and how to parallelise the different imputations across cores, so I would like to use this package if it is feasible.

My questions are:

1) How large a data set can MICE handle?

2) Does any body have experience imputing large data sets using the MICE package, if so how large was it and how long did it take to run?

3) Are there any other packages which do the same thing but designed specifically to handle large data set?


closed as off-topic by rolando2, kjetil b halvorsen, mdewey, Michael Chernick, Peter Flom Nov 23 '18 at 10:30

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – rolando2, kjetil b halvorsen, mdewey, Michael Chernick, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ I assume you mean a subset of observations, not variables? You might need to ask on an R specific site for other package suggestions. $\endgroup$ – mdewey Dec 7 '17 at 13:33
  • $\begingroup$ Mice can feel like a black box. I'd write my own algo so as to avoid having to wonder whether it is stuck $\endgroup$ – generic_user Dec 7 '17 at 13:45
  • $\begingroup$ In terms of how large a data set, R is more or less limited to how much memory (RAM) your computer/server has to store the data that you're working with in the terminal/gui. There are work methods to use the Computer's hard drive as well if you run out of RAM, but I've never use them myself. Can't comment on the speed as I'm not sure the exact computations people are running. Might have to check if anyone's posted blog or how-to's on the package usage and their system time to compute certain things. It'll be dependent on data size and CPU speed. $\endgroup$ – Kunio Dec 7 '17 at 17:59