0
$\begingroup$

I wonder why preProcess function from R's caret package used for imputation of dataset's missing values returns less observations than in original dataset?
For example:

library(caret)

t <- data.frame(seq_len(100000),seq_len(100000))

for (i in 1:100000) 
{
if (i %% 10 == 0) t[i,1] <- NA; 
if (i %% 100 == 0) t[i,2] <- NA 
}

preProcValues <- preProcess(t, method = c("knnImpute"))

preProcValues will contain only 90000 observations of 2 variables while 100000 is expected.

$\endgroup$
4
$\begingroup$

preProcess does not return values, it simply sets up the whole imputation model based on the provided data. So, you need to run predict (requiring also the RANN package), but even if you do so with your artificial data you'll get an error:

Error in FUN(newX[, i], ...) : cannot impute when all predictors are missing in the new data point

as the imputation can not work in rows where both your predictors are NA's.

Here's a demonstration with only 20 rows, for clarity and easy inspection:

library(caret)

t <- data.frame(seq_len(20),seq_len(20))

for (i in 1:20) 
{
  if (i %% 3 == 0) t[i,1] <- NA; 
  if (i %% 7 == 0) t[i,2] <- NA 
}

names(t) <- c('V1', 'V2')

preProcValues <- preProcess(t, method = c("knnImpute"))

library(RANN)

t_imp <- predict(preProcValues, t)

When viewing the result, keep in mind that methods "center", "scale" have been automaticaly added to your preprocessing, even if you did not invoke them explicitly:

> str(preProcValues)
List of 19
$ call      : language preProcess.default(x = t, method = c("knnImpute"))
$ dim       : int [1:2] 12 2
$ bc        : NULL
$ yj        : NULL
$ et        : NULL
$ mean      : Named num [1:2] 10.5 10.5
 ..- attr(*, "names")= chr [1:2] "V1" "V2"
$ std       : Named num [1:2] 6.25 6.14
 ..- attr(*, "names")= chr [1:2] "V1" "V2"
$ ranges    : NULL
$ rotation  : NULL
$ method    : chr [1:3] "knnImpute" "scale" "center"
$ thresh    : num 0.95
$ pcaComp   : NULL
$ numComp   : NULL
$ ica       : NULL
$ k         : num 5
$ knnSummary:function (x, ...)  
$ bagImp    : NULL
$ median    : NULL
$ data      : num [1:12, 1:2] -1.434 -1.283 -0.981 -0.83 -0.377 ...
 ..- attr(*, "dimnames")=List of 2
     .. ..$ : chr [1:12] "1" "2" "4" "5" ...
 .. ..$ : chr [1:2] "V1" "V2"
 ..- attr(*, "scaled:center")= Named num [1:2] 10.5 10.5
 .. ..- attr(*, "names")= chr [1:2] "V1" "V2"
 ..- attr(*, "scaled:scale")= Named num [1:2] 6.63 6.63
 .. ..- attr(*, "names")= chr [1:2] "V1" "V2"
- attr(*, "class")= chr "preProcess"
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Ok, I think I finally got the idea. predict is used to get the imputed data, not the target variable based on imputed data. $\endgroup$ – truf Oct 24 '15 at 22:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.