PCA and k-fold cross-validation in caret package in R

Question

I just re-watched a lecture from the Machine Learning course on Coursera. In the section where the professor discusses PCA for pre-processing data in supervised learning applications, he says PCA should only be performed on the training data and then the mapping is used to transform the cross validation and test sets. See also PCA and the train/test split.

However, in the caret R package, the training data you pass into the train() function is already processed by PCA. So when the algorithm performs k-fold cross validation, the cross validation set has already been processed with PCA via preProcess() and predict() and is in fact used in the PCA "fitting".

Is my understanding of the situation correct? I.e. caret's procedure for cross-validation with PCA (or in fact, with any rescaling/centering method) is "wrong" because the pre-processing of data is performed on the cross-validation set as well as the training set? And if so, how big an impact would this have on the results?

Answer 1

I didn't see the lecture, so I can't comment on what was said.

My $0.02: If you want to get good estimates of performance using resampling, you should really do all of the operations during resampling instead of prior. This is really true of feature selection [1] as well as non-trivial operations like PCA. If it adds uncertainty to the results, include it in resampling.

Think about principal component regression: PCA followed by linear regression on some of the components. PCA estimates parameters (with noise) and the number of components must also be chosen (different values will result in different results => more noise).

Say we used 10 fold CV with scheme 1:

conduct PCA
pick the number of components
for each fold:
   split data
   fit linear regression on the 90% used for training
   predict the 10% held out
end:

or scheme 2:

for each fold:
   split data
   conduct PCA on the 90% used for training
   pick the number of components
   fit linear regression
   predict the 10% held out
end:

It should be clear than the second approach should produce error estimates that reflect the uncertainty caused by PCA, selection of the number of components and the linear regression. In effect, the CV in the first scheme has no idea what preceded it.

I'm guilty of not always doing all the operations w/in resampling, but only when I don't really care about performance estimates (which is unusual).

Is there much difference between the two schemes? It depends on the data and the pre-processing. If you are only centering and scaling, probably not. If you have a ton of data, probably not. As the training set size goes down, the risk of getting poor estimates goes up, especially if n is close to p.

I can say with certainty from experience that not including supervised feature selection within resampling is a really bad idea (without large training sets). I don't see why pre-processing would be immune to this (to some degree).

@mchangun: I think that the number of components is a tuning parameter and you would probably want to pick it using performance estimates that are generalizable. You could automatically pick K such that at least X% of the variance is explained and include that process within resampling so we account for the noise in that process.

Max

[1] Ambroise, C., & McLachlan, G. (2002). Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences, 99(10), 6562–6566.

Answer 2

Don't preProcess the data prior to running the train function! Use the preProcess argument for the train function, and the pre-processing will be applied to each re-sampling iteration.

e.g. don't do this:

library(caret)
dat <- iris
pp <- preProcess(dat[,-5], method="pca")
dat[,-5] <- predict(pp, dat[,-5])
knnFit1 <- train(Species~., dat, method = "knn", preProcess=c("pca"), 
    trControl = trainControl(method = "cv"))

do this!

dat <- iris
knnFit2 <- train(Species~., dat, method = "knn", preProcess=c("pca"), 
    trControl = trainControl(method = "cv"))