0
$\begingroup$

I would like to make a PC/PLS regression and assess the resulting model's predictive power. The strategy is the classical splitting into training/validation/test sets, and using training/validation sets to fit the model and test set to assess the prediction performance.

My question is, can I use the whole data set to calculate the PCA if I then use only training & validation sets to fit the model parameters (i.e., regression weights and number of PCA components)?

Performing PCA on the whole data set would mean peek at the test set and might be considered cheating. However, if the data set is small one might not get an accurate estimate of the statistical structure in the data unless the whole set is considered, in which case it could be seen as legitimate (?)

I couldn't find any discussion of this over the web, so any pointers would be appreciated.

Thanks!

$\endgroup$
1
$\begingroup$

can I use the whole data set to calculate the PCA if I then use only training & validation sets to fit the model parameters (i.e., regression weights and number of PCA components)?

No, this may lead to very overoptimistic test results.

For the data sets I typically work with (spectroscopic data sets, wide matrices wrt. the number of independent specimen/patients; classification) I've seen the number of misclassifications being underestimated by an order of magnitude when using the whole data set to calculate the loadings.

I think of it this way: the PCA/PLS projection reduces (for my data) 100 - 1000 variates to maybe 10, i.e. 10 $\times$ 100 - 1000 coefficients are calculated. The regression (or in my case classification) calculates only maybe 10 more coefficients. Thus, I'd intuitively expect that the more crucial (and difficult) step is to get the loadings. Thus, independence of test and training data is crucial for the step of loading calculation / validation of the calculated loadings.

Here's my recommendation: do it both ways and have a look at the difference. Maybe even report the results here.

$\endgroup$
  • $\begingroup$ Hi! Thanks a lot for your answer. I like your argument - thinking of it that way, it seems obvious indeed that getting the loadings is the most important step. I've already tried doing it both ways: not surprisingly, calculating the loadings on the whole data sets yields slightly better results. The reason I'm asking the question is that it seems to me that both ways of doing it could be justified, and because I haven't been able to find a formal discussion of this topic, I'm wondering what other people would think of it. $\endgroup$ – wawrzeniec Jun 15 '15 at 1:39
  • $\begingroup$ In the case of PLS, calculating the loadings involves the dependent variable, which clearly violates independence of the test set. In PCR, it involve only the predictors, so I can't see why it is wrong, although it would likely not be considered best practice. I'm happy to report the results here as soon as I can find the time. $\endgroup$ – wawrzeniec Jun 15 '15 at 2:02
  • $\begingroup$ @wawrzeniec: The independent variables (don't be mislead by "independent" here, it does not mean these are independent in the way you need for independent test cases) also contain information about the problem. As you say, the PCA may benefit from including these cases. This is a very strong point that testing on cases that did not enter the model calculations in any way (not for the centering, not for the loadings, not for the regression, not for model selection, etc.) is needed. $\endgroup$ – cbeleites supports Monica Jun 15 '15 at 7:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.