# Missing value imputation in huge dataset

I have a huge data (4M x 17) that has missing values. Two columns are categorical, rest all are numerical. Given the huge amount of data, running any imputation method runs forever. What should I do?

I was wondering if I could train some model using a subset of data and then use that to impute values in the full data. For example, if I were to use MICE package, I would like something like following to exist:

> testMice <- mice(myData[1:100000,]) # runs fine
> testTot <- predict(testMice, myData) # hypothetical


Running the imputation on whole dataset is computationally expensive, so I would run it on only the first 100K observations. Then I would try to use the output to impute the whole data.

• Why do you say running imputation on this dataset takes forever? A 4 million by 17 size dataset is quite small in my opinion and there is no reason you shouldn't be able to run any number of imputation methods on this dataset. – StatsStudent Jul 16 '16 at 5:08
• I ran mice. It ran for hours – Sonu Mishra Jul 16 '16 at 5:09
• If you don't want to do what I've suggested above, what you are suggesting should be okay. Before you run your analysis, be sure you take a random sampling of your data instead of grabbing just the first 100K observations in case there is some order effect that will affect the imputations. – StatsStudent Jul 16 '16 at 5:20
• Also have you seen this? stats.stackexchange.com/questions/100020/… – StatsStudent Jul 16 '16 at 5:21
• One last thought after looking at the MICE documentation briefly. Try changing the fastppm option to norm. This should speed completion of the imputation substantially. – StatsStudent Jul 16 '16 at 5:25