# Fast missing data imputation in R for big data that is more sophisticated than simply imputing the means?

I need a package for missing data imputation in R. But since I am dealing with big data, the number of missing data entries can also be high. The packages which impute using mean or median are of course working fast, but more complicated packages which impute using regression or PCA take too long for a high number of missing values. I tried missMDA and missForest, but as I said, they look like taking forever. There is a package named FastImputation, but I could not figure out how to use it when I have no patterns from some training data. Any suggestions of packages which would impute fast?

• How big is your big data? – Ellis Valentiner May 20 '14 at 2:28
• It is gene expression matrix of ~10000 genes as columns and a few hundred patients as rows. And the number of missing values can be even 80% of all matrix entries – user5054 May 20 '14 at 5:16
• Playing with the 'threshold' and 'maxiter' parameters of 'imputePCA' in 'missMDA' package helps a lot for fastening the imputation performed by 'imputePCA' (of course if you do not require very high accuracy). – user5054 Aug 31 '14 at 7:42
• What do you want to analyze that you really need imputation? Why does available cases analysis not work for you? – Horst Grünbusch Jan 17 '15 at 18:33

I used mice (multiple imputation by chained equation). It's fairly fast, and quite simple. I used it on 3000 obs. for c.a. 10 variables. Done in 10min on an old computer. Further, I believe it is one of the best multiple-imputation packages out there. It can use regression to impute, among other methods.

You need to create a dataframe with the variable you want to impute, and include every variable that might predict values of that variable (so every var. in your model + possibly other var. as well). The mice package will impute every missing value in that dataframe.

Simplest way of imputing. Gives you a dataframe Datimp that has five imputed data + the original data.

library(mice)
#m=5 number of multiple imputations
#maxit=10 number of iterations. 10-20 is sufficient.
imp <- mice(Dat1, m=5, maxit=10, printFlag=TRUE)
Datimp <- complete(imp, "long", include=TRUE)
write.table(Datimp, "C:/.../impute1.txt",
sep="\t", dec=",", row.names=FALSE)

A better way to do this is:

library(mice)
Dat1 <- subset(Dat, select=c(id, faculty, gender, age, job, salary)) #create subset
#of variables you would like to either impute or use as predictors for imputation.
ini <- mice(Dat1, maxit=0, pri=F)
pred <- ini$pred pred[,c("id", "faculty")] <- 0 #variables you do not want to use as predictors (but #want to have in the dataset, can't add them later. meth <- ini$meth
meth[c("id", "faculty", "gender", "age", "job")] <- "" #choose a prediction method
#for imputing your variables. Here I don't want these variables to be imputed, so I
#choose "" (empty, no mehod).
imp <- mice(Dat1, m=5, maxit=10, printFlag=TRUE, pred=pred, meth=meth, seed=2345)
Datimp <- complete(imp, "long", include=TRUE)
write.table(Datimp, "C:/.../impute1.txt",
sep="\t", dec=",", row.names=FALSE)

See if your imputations were any good:

library(lattice)
com <- complete(imp, "long", inc=T)
col <- rep(c("blue","red")[1+as.numeric(is.na(imp$salary))],6) stripplot(salary~.imp, data=com, jit=TRUE, fac=0.8, col=col, pch=20, xlab="Imputation number",cex=0.25) densityplot(salary~.imp, data=com, jit=TRUE, fac=0.8, col=col, pch=20, xlab="Imputation number",cex=0.25) long <- complete(imp,"long") levels(long$.imp) <- paste("Imputation",1:22)
long <- cbind(long, salary.na=is.na(imp$data$salary))
densityplot(~salary|.imp, data=long, group=salary, plot.points=FALSE, ref=TRUE,
xlab="Salary",scales=list(y=list(draw=F)),
par.settings=simpleTheme(col.line=rep(c("blue","red"))), auto.key =
list(columns=2,text=c("Observed","Imputed")))

Finally, and importantly. You can't just save your new dataset and use your imputed values as normal observed values. You use pooled regression or pooled lmer ...So the uncertainty of the imputed values is taken into account.

fit1 <- with(imp, lm(salary ~ gender, na.action=na.omit))
summary(est <- pool(fit1))