I have a huge data (4M x 17) that has missing values. Two columns are categorical, rest all are numerical. Given the huge amount of data, running any imputation method runs forever. What should I do?

I was wondering if I could train some model using a subset of data and then use that to impute values in the full data. For example, if I were to use MICE package, I would like something like following to exist:

> testMice <- mice(myData[1:100000,]) # runs fine  
> testTot <- predict(testMice, myData) # hypothetical

Running the imputation on whole dataset is computationally expensive, so I would run it on only the first 100K observations. Then I would try to use the output to impute the whole data.

  • 1
    $\begingroup$ Why do you say running imputation on this dataset takes forever? A 4 million by 17 size dataset is quite small in my opinion and there is no reason you shouldn't be able to run any number of imputation methods on this dataset. $\endgroup$ – StatsStudent Jul 16 '16 at 5:08
  • $\begingroup$ I ran mice. It ran for hours $\endgroup$ – Sonu Mishra Jul 16 '16 at 5:09
  • 3
    $\begingroup$ If you don't want to do what I've suggested above, what you are suggesting should be okay. Before you run your analysis, be sure you take a random sampling of your data instead of grabbing just the first 100K observations in case there is some order effect that will affect the imputations. $\endgroup$ – StatsStudent Jul 16 '16 at 5:20
  • 1
    $\begingroup$ Also have you seen this? stats.stackexchange.com/questions/100020/… $\endgroup$ – StatsStudent Jul 16 '16 at 5:21
  • 1
    $\begingroup$ One last thought after looking at the MICE documentation briefly. Try changing the fastppm option to norm. This should speed completion of the imputation substantially. $\endgroup$ – StatsStudent Jul 16 '16 at 5:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.