10
$\begingroup$

My question in short: are there methods to improve on the running time of R MICE (data imputation)?

I'm working with a data set (30 variables, 1.3 million rows) which contains (quite randomly) missing data. About 8% of the observations in about 15 out of 30 variables contain NAs. In order to impute the missing data, I'm running the MICE function, part of the MICE package.

I experience quite slow running time, even on a subset (100,000 rows), with method="fastpmm" and m=1 and runs for about 15 minutes.

Is there a way to improve on running time without losing too much in performance? (mice.impute.mean is quite fast, but comes with important loss of information!).

Reproducible code:

library(mice)
df <- data.frame(replicate(30,sample(c(NA,1:10),1000000,rep=TRUE)))
df <- data.frame(scale(df))

output <- mice(df, m=1, method = "fastpmm")
Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ In general: are these kind of questions appropriate on Cross Validated, or better suited for Stack Overflow? $\endgroup$ – Dendrobates Apr 28 '16 at 14:01
  • 3
    $\begingroup$ It might be a judgment call. Because (as a general principle) the most substantial improvements in running times are obtained by understanding the underlying algorithms, I would expect your best chance of getting a really effective answer might be here, in a community where people might be able to suggest alternative approaches. If you don't get adequate answers in a day or two, then just flag this post for migration and we'll send it on to SO (along with any answers and comments it may have collected in the meantime). $\endgroup$ – whuber♦ Apr 28 '16 at 15:22
  • 1
    $\begingroup$ You can change 'fastppm' option to 'norm', it is going to be faster $\endgroup$ – marc1s Apr 28 '16 at 16:16
  • 1
    $\begingroup$ Thx @marc1s, that did improve a lot for large datasets. For a random data frame (as above) with 10,000 rows, the method "norm" was about 4 times faster than "fastpmm". With 50,000 rows it was even 12 times faster. Hence, the relative gain in running time is increasing by the number of rows. $\endgroup$ – Dendrobates Apr 29 '16 at 7:49
  • 1
    $\begingroup$ Depending on the model you are going to run, it might be faster to use maximum likelihood (or full information maximum likelihood) estimation, which is asymptotically equivalent to imputation if the model is correctly specified. Here's a paper I was involved in, that compares the different methods: emeraldinsight.com/doi/abs/10.1108/JCP-02-2015-0007 $\endgroup$ – Jeremy Miles Dec 28 '16 at 17:50
5
$\begingroup$

You can use quickpred() from mice package using which you can limit the predictors by specifying the mincor (Minimum correlation) and minpuc (proportion of usable cases). Also you can use the exclude and include parameters for controlling the predictors.

Improve this answer
$\endgroup$
2
$\begingroup$

I made a wrapper for the mice function that includes one extra argument, droplist, where you can pass a character vector of predictor variables that you do not want used in the right-hand-side of the imputation formulas. This was for speed, as I found that factor variables with many levels would slow down the imputation considerably. I wasn't aware of the quickpred function referenced by @Aanish, and perhaps you could use both concepts together.

Below is the function as it appears in my glmmplus package. If you find it useful, I may open a pull request in the actual mice package.

ImputeData <- function(data, m = 10, maxit = 15, droplist = NULL) {
  if (length(intersect(names(data), droplist)) < length(droplist)) {
    stop("Droplist variables not found in data set")
  }
  predictorMatrix <- (1 - diag(1, ncol(data)))
  for (term in droplist) {
  drop.index <- which(names(data) == term)
    predictorMatrix[, drop.index] <- 0
  }
  mids.out <- mice(data, m = m, maxit = maxit,
                   predictorMatrix = predictorMatrix)
  return(mids.out)
}
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.