2
$\begingroup$

I am planning to carry out multiple imputation on a large data set with around 4,000,000 observations and 32 variables and 10 interactions. Of these variables six have missing data which I need to impute.

I plan to use MICE package from R and the fully conditionally specified method, outlined here: https://www.jstatsoft.org/article/view/v045i03

I have tried to run a test imputation using a subset of 10,000 observations, for 1 iteration and 5 imputation data sets and it is taking very long to run, so I am worried how long it will take when I try it with the full data set. I have spent quite a lot of time properly understanding how this package works and how to parallelise the different imputations across cores, so I would like to use this package if it is feasible.

My questions are:

1) How large a data set can MICE handle?

2) Does any body have experience imputing large data sets using the MICE package, if so how large was it and how long did it take to run?

3) Are there any other packages which do the same thing but designed specifically to handle large data set?

$\endgroup$
3
  • $\begingroup$ I assume you mean a subset of observations, not variables? You might need to ask on an R specific site for other package suggestions. $\endgroup$
    – mdewey
    Dec 7 '17 at 13:33
  • $\begingroup$ Mice can feel like a black box. I'd write my own algo so as to avoid having to wonder whether it is stuck $\endgroup$ Dec 7 '17 at 13:45
  • $\begingroup$ In terms of how large a data set, R is more or less limited to how much memory (RAM) your computer/server has to store the data that you're working with in the terminal/gui. There are work methods to use the Computer's hard drive as well if you run out of RAM, but I've never use them myself. Can't comment on the speed as I'm not sure the exact computations people are running. Might have to check if anyone's posted blog or how-to's on the package usage and their system time to compute certain things. It'll be dependent on data size and CPU speed. $\endgroup$
    – Kunio
    Dec 7 '17 at 17:59