6
$\begingroup$

I need a package for missing data imputation in R. But since I am dealing with big data, the number of missing data entries can also be high. The packages which impute using mean or median are of course working fast, but more complicated packages which impute using regression or PCA take too long for a high number of missing values. I tried missMDA and missForest, but as I said, they look like taking forever. There is a package named FastImputation, but I could not figure out how to use it when I have no patterns from some training data. Any suggestions of packages which would impute fast?

$\endgroup$
  • 5
    $\begingroup$ How big is your big data? $\endgroup$ – Ellis Valentiner May 20 '14 at 2:28
  • $\begingroup$ It is gene expression matrix of ~10000 genes as columns and a few hundred patients as rows. And the number of missing values can be even 80% of all matrix entries $\endgroup$ – user5054 May 20 '14 at 5:16
  • $\begingroup$ Playing with the 'threshold' and 'maxiter' parameters of 'imputePCA' in 'missMDA' package helps a lot for fastening the imputation performed by 'imputePCA' (of course if you do not require very high accuracy). $\endgroup$ – user5054 Aug 31 '14 at 7:42
  • $\begingroup$ What do you want to analyze that you really need imputation? Why does available cases analysis not work for you? $\endgroup$ – Horst Grünbusch Jan 17 '15 at 18:33
9
$\begingroup$

I used mice (multiple imputation by chained equation). It's fairly fast, and quite simple. I used it on 3000 obs. for c.a. 10 variables. Done in 10min on an old computer. Further, I believe it is one of the best multiple-imputation packages out there. It can use regression to impute, among other methods.

You need to create a dataframe with the variable you want to impute, and include every variable that might predict values of that variable (so every var. in your model + possibly other var. as well). The mice package will impute every missing value in that dataframe.

Simplest way of imputing. Gives you a dataframe Datimp that has five imputed data + the original data.

library(mice)
#m=5 number of multiple imputations
#maxit=10 number of iterations. 10-20 is sufficient.
imp <- mice(Dat1, m=5, maxit=10, printFlag=TRUE) 
Datimp <- complete(imp, "long", include=TRUE)
write.table(Datimp, "C:/.../impute1.txt",
            sep="\t", dec=",", row.names=FALSE)

A better way to do this is:

library(mice)
Dat1 <- subset(Dat, select=c(id, faculty, gender, age, job, salary)) #create subset
#of variables you would like to either impute or use as predictors for imputation.
ini <- mice(Dat1, maxit=0, pri=F)
pred <- ini$pred
    pred[,c("id", "faculty")] <- 0 #variables you do not want to use as predictors (but
    #want to have in the dataset, can't add them later.
    meth <- ini$meth
meth[c("id", "faculty", "gender", "age", "job")] <- "" #choose a prediction method
#for imputing your variables. Here I don't want these variables to be imputed, so I
#choose "" (empty, no mehod).
imp <- mice(Dat1, m=5, maxit=10, printFlag=TRUE, pred=pred, meth=meth, seed=2345) 
Datimp <- complete(imp, "long", include=TRUE)
write.table(Datimp, "C:/.../impute1.txt",
            sep="\t", dec=",", row.names=FALSE)

See if your imputations were any good:

library(lattice)
com <- complete(imp, "long", inc=T)
col <- rep(c("blue","red")[1+as.numeric(is.na(imp$salary))],6)
stripplot(salary~.imp, data=com, jit=TRUE, fac=0.8, col=col, pch=20,
xlab="Imputation number",cex=0.25) 
densityplot(salary~.imp, data=com, jit=TRUE, fac=0.8, col=col, pch=20,
xlab="Imputation number",cex=0.25) 

long <- complete(imp,"long")
levels(long$.imp) <- paste("Imputation",1:22)
    long <- cbind(long, salary.na=is.na(imp$data$salary))
densityplot(~salary|.imp, data=long, group=salary, plot.points=FALSE, ref=TRUE, 
xlab="Salary",scales=list(y=list(draw=F)),
par.settings=simpleTheme(col.line=rep(c("blue","red"))), auto.key =
list(columns=2,text=c("Observed","Imputed"))) 

Finally, and importantly. You can't just save your new dataset and use your imputed values as normal observed values. You use pooled regression or pooled lmer ...So the uncertainty of the imputed values is taken into account.

fit1 <- with(imp, lm(salary ~ gender, na.action=na.omit))
summary(est <- pool(fit1))
pool.r.squared(fit1,adjusted=FALSE)
$\endgroup$
  • 2
    $\begingroup$ Actually, mice allows you to specify how missing values are filled in -- simple regression is but one of many methods. Also, your code chunk is inaccurate -- by specifying complete(imp), you are only returning the first of your 5 imputations. $\endgroup$ – Patrick S. Forscher Jan 17 '15 at 22:08
  • 1
    $\begingroup$ Yes, you are right. I have improved my code since I wrote this answer. If I remember correctly, the command is long(imp) to get all the imputations Datimp <- long(imp, include = TRUE) will include the original dataset with the imputed datasets. $\endgroup$ – Helgi Guðmundsson Jan 18 '15 at 11:25
  • $\begingroup$ If you have improved your code and wish to share your improvements, there is an "edit" button on the lower left-hand corner of your answer. $\endgroup$ – Patrick S. Forscher Jan 18 '15 at 15:09
0
$\begingroup$

The Hmisc package can probably do this with the imputation function (aregImpute). Agree that columns are plenty, but rows are few. Should probably be handled by Hisc...

$\endgroup$
  • 1
    $\begingroup$ Can you expound on this and provide an example of using the Hmisc package to do this? $\endgroup$ – coip Mar 28 '18 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.