16
$\begingroup$

I just re-watched a lecture from the Machine Learning course on Coursera. In the section where the professor discusses PCA for pre-processing data in supervised learning applications, he says PCA should only be performed on the training data and then the mapping is used to transform the cross validation and test sets. See also PCA and the train/test split.

However, in the caret R package, the training data you pass into the train() function is already processed by PCA. So when the algorithm performs k-fold cross validation, the cross validation set has already been processed with PCA via preProcess() and predict() and is in fact used in the PCA "fitting".

Is my understanding of the situation correct? I.e. caret's procedure for cross-validation with PCA (or in fact, with any rescaling/centering method) is "wrong" because the pre-processing of data is performed on the cross-validation set as well as the training set? And if so, how big an impact would this have on the results?

$\endgroup$
  • 1
    $\begingroup$ Interesting question. Does Professor Ng give reasons why "PCA should only be performed on the training data and then the mapping is used to transform the cross validation and test sets"? My intuition is that the impact should be minimal, as train/cv/test data are assumed to come from the same distribution, and PCA does not involve the output/dependent variable(s). $\endgroup$ – miura Dec 19 '12 at 8:22
  • 1
    $\begingroup$ @miura Yes he does. When performing PCA, we need to solve for the minimum number k dimensions in the output vectors needed to achieve the desired "retained variance". This parameter k is now in effect (in my understanding) an extra parameter of our algorithm that we need to fit. If we perform PCA on the cross-validation set as well, we are in effect training a portion of our algorithm on the CV set, thus diminishing the value of cross-validation. $\endgroup$ – mchangun Dec 19 '12 at 9:17
  • $\begingroup$ I think you are confusing PCA and MDS but in both methods you don't set a desired retained variance. But you can calculate it from the ajustment they bring to reduce to k dimensions. You don't need to fit it because the first dimension will always reduce more variance than the second one, and the second one more than the third... $\endgroup$ – llrs Nov 10 '16 at 17:00
18
$\begingroup$

I didn't see the lecture, so I can't comment on what was said.

My $0.02: If you want to get good estimates of performance using resampling, you should really do all of the operations during resampling instead of prior. This is really true of feature selection [1] as well as non-trivial operations like PCA. If it adds uncertainty to the results, include it in resampling.

Think about principal component regression: PCA followed by linear regression on some of the components. PCA estimates parameters (with noise) and the number of components must also be chosen (different values will result in different results => more noise).

Say we used 10 fold CV with scheme 1:

conduct PCA
pick the number of components
for each fold:
   split data
   fit linear regression on the 90% used for training
   predict the 10% held out
end:

or scheme 2:

for each fold:
   split data
   conduct PCA on the 90% used for training
   pick the number of components
   fit linear regression
   predict the 10% held out
end:

It should be clear than the second approach should produce error estimates that reflect the uncertainty caused by PCA, selection of the number of components and the linear regression. In effect, the CV in the first scheme has no idea what preceded it.

I'm guilty of not always doing all the operations w/in resampling, but only when I don't really care about performance estimates (which is unusual).

Is there much difference between the two schemes? It depends on the data and the pre-processing. If you are only centering and scaling, probably not. If you have a ton of data, probably not. As the training set size goes down, the risk of getting poor estimates goes up, especially if n is close to p.

I can say with certainty from experience that not including supervised feature selection within resampling is a really bad idea (without large training sets). I don't see why pre-processing would be immune to this (to some degree).

@mchangun: I think that the number of components is a tuning parameter and you would probably want to pick it using performance estimates that are generalizable. You could automatically pick K such that at least X% of the variance is explained and include that process within resampling so we account for the noise in that process.

Max

[1] Ambroise, C., & McLachlan, G. (2002). Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences, 99(10), 6562–6566.

$\endgroup$
21
$\begingroup$

Don't preProcess the data prior to running the train function! Use the preProcess argument for the train function, and the pre-processing will be applied to each re-sampling iteration.

e.g. don't do this:

library(caret)
dat <- iris
pp <- preProcess(dat[,-5], method="pca")
dat[,-5] <- predict(pp, dat[,-5])
knnFit1 <- train(Species~., dat, method = "knn", preProcess=c("pca"), 
    trControl = trainControl(method = "cv"))

do this!

dat <- iris
knnFit2 <- train(Species~., dat, method = "knn", preProcess=c("pca"), 
    trControl = trainControl(method = "cv"))
$\endgroup$
  • 1
    $\begingroup$ If I do the prProcess as you suggest, do I need to preProcess new data when I want to use the model for predictions? At the moment, I only do: pred <- predict(knnFit2, newdata) where newdata is not scaled. Is Caret smart enough to know that it needs to preProcess newdata before using it for predictions? $\endgroup$ – mchangun Jan 10 '13 at 10:38
  • 7
    $\begingroup$ @mchangun no. caret saves the pre-processing parameters, and will pre-process the new data in the predict.caret call. it's pretty slick! $\endgroup$ – Zach Sep 9 '13 at 2:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.