7
$\begingroup$

I need a package for missing data imputation in R. But since I am dealing with big data, the number of missing data entries can also be high. The packages which impute using mean or median are of course working fast, but more complicated packages which impute using regression or PCA take too long for a high number of missing values. I tried missMDA and missForest, but as I said, they look like taking forever. There is a package named FastImputation, but I could not figure out how to use it when I have no patterns from some training data. Any suggestions of packages which would impute fast?

$\endgroup$
4
  • 5
    $\begingroup$ How big is your big data? $\endgroup$ May 20, 2014 at 2:28
  • $\begingroup$ It is gene expression matrix of ~10000 genes as columns and a few hundred patients as rows. And the number of missing values can be even 80% of all matrix entries $\endgroup$
    – user5054
    May 20, 2014 at 5:16
  • $\begingroup$ Playing with the 'threshold' and 'maxiter' parameters of 'imputePCA' in 'missMDA' package helps a lot for fastening the imputation performed by 'imputePCA' (of course if you do not require very high accuracy). $\endgroup$
    – user5054
    Aug 31, 2014 at 7:42
  • $\begingroup$ What do you want to analyze that you really need imputation? Why does available cases analysis not work for you? $\endgroup$ Jan 17, 2015 at 18:33

2 Answers 2

10
$\begingroup$

I used mice (multiple imputation by chained equation). It's fairly fast, and quite simple. I used it on 3000 obs. for c.a. 10 variables. Done in 10min on an old computer. Further, I believe it is one of the best multiple-imputation packages out there. It can use regression to impute, among other methods.

You need to create a dataframe with the variable you want to impute, and include every variable that might predict values of that variable (so every var. in your model + possibly other var. as well). The mice package will impute every missing value in that dataframe.

Simplest way of imputing. Gives you a dataframe Datimp that has five imputed data + the original data.

library(mice)
#m=5 number of multiple imputations
#maxit=10 number of iterations. 10-20 is sufficient.
imp <- mice(Dat1, m=5, maxit=10, printFlag=TRUE) 
Datimp <- complete(imp, "long", include=TRUE)
write.table(Datimp, "C:/.../impute1.txt",
            sep="\t", dec=",", row.names=FALSE)

A better way to do this is:

library(mice)
Dat1 <- subset(Dat, select=c(id, faculty, gender, age, job, salary)) #create subset
#of variables you would like to either impute or use as predictors for imputation.
ini <- mice(Dat1, maxit=0, pri=F)
pred <- ini$pred
    pred[,c("id", "faculty")] <- 0 #variables you do not want to use as predictors (but
    #want to have in the dataset, can't add them later.
    meth <- ini$meth
meth[c("id", "faculty", "gender", "age", "job")] <- "" #choose a prediction method
#for imputing your variables. Here I don't want these variables to be imputed, so I
#choose "" (empty, no mehod).
imp <- mice(Dat1, m=5, maxit=10, printFlag=TRUE, pred=pred, meth=meth, seed=2345) 
Datimp <- complete(imp, "long", include=TRUE)
write.table(Datimp, "C:/.../impute1.txt",
            sep="\t", dec=",", row.names=FALSE)

See if your imputations were any good:

library(lattice)
com <- complete(imp, "long", inc=T)
col <- rep(c("blue","red")[1+as.numeric(is.na(imp$salary))],6)
stripplot(salary~.imp, data=com, jit=TRUE, fac=0.8, col=col, pch=20,
xlab="Imputation number",cex=0.25) 
densityplot(salary~.imp, data=com, jit=TRUE, fac=0.8, col=col, pch=20,
xlab="Imputation number",cex=0.25) 

long <- complete(imp,"long")
levels(long$.imp) <- paste("Imputation",1:22)
    long <- cbind(long, salary.na=is.na(imp$data$salary))
densityplot(~salary|.imp, data=long, group=salary, plot.points=FALSE, ref=TRUE, 
xlab="Salary",scales=list(y=list(draw=F)),
par.settings=simpleTheme(col.line=rep(c("blue","red"))), auto.key =
list(columns=2,text=c("Observed","Imputed"))) 

Finally, and importantly. You can't just save your new dataset and use your imputed values as normal observed values. You use pooled regression or pooled lmer ...So the uncertainty of the imputed values is taken into account.

fit1 <- with(imp, lm(salary ~ gender, na.action=na.omit))
summary(est <- pool(fit1))
pool.r.squared(fit1,adjusted=FALSE)
$\endgroup$
3
  • 2
    $\begingroup$ Actually, mice allows you to specify how missing values are filled in -- simple regression is but one of many methods. Also, your code chunk is inaccurate -- by specifying complete(imp), you are only returning the first of your 5 imputations. $\endgroup$ Jan 17, 2015 at 22:08
  • 1
    $\begingroup$ Yes, you are right. I have improved my code since I wrote this answer. If I remember correctly, the command is long(imp) to get all the imputations Datimp <- long(imp, include = TRUE) will include the original dataset with the imputed datasets. $\endgroup$ Jan 18, 2015 at 11:25
  • 1
    $\begingroup$ If you have improved your code and wish to share your improvements, there is an "edit" button on the lower left-hand corner of your answer. $\endgroup$ Jan 18, 2015 at 15:09
0
$\begingroup$

The Hmisc package can probably do this with the imputation function (aregImpute). Agree that columns are plenty, but rows are few. Should probably be handled by Hisc...

$\endgroup$
1
  • 1
    $\begingroup$ Can you expound on this and provide an example of using the Hmisc package to do this? $\endgroup$
    – coip
    Mar 28, 2018 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.