4
$\begingroup$

I'm currently looking through Chapter 6 of An Introduction to Statistical Leanring by Gareth James. I am working through the Chapter 6 lab regarding Principal Component Analysis.

In the lab we first use k-fold cross-validation with 10 folds in order to find the optimal number of principal components to use. There are 19 components in total, and cross-validation shows that the lowest average MSE across all folds is achieved for 18 components, although 5 ish seems to capture most of the variance.

This all makes sense to me, but the next stage of the lab is slightly confusing. The book then goes on to say "We now perform PCR on the training data and evaluate its test set performance."

The data is now split into a single training and test set (split 50% between the two), and the 10 fold cross-validation process is repeated, this time revealing that 6 principal components gives the lowest training MSE. The lab then calculated the test MSE for a linear regression model containing these 6 principal components.

The final stage doesn't make sense to me. Why, once performing the initial cross-validation do we repeat the process on only half the data-set? Is it because, although a training-test split was used in the first cross-validation process, there was some overlap between the training and test splits. Is the purpose of the repetition of the cross-validation to be able to calculate a test MSE which is completely independent of the training data?

And as a final question, is the reason the lowest training MSEs for the first and second cross-validations different because less of the data was used in the second instance?

$\endgroup$

1 Answer 1

1
$\begingroup$

Why, once performing the initial cross-validation do we repeat the process on only half the data-set? Is it because, although a training-test split was used in the first cross-validation process, there was some overlap between the training and test splits. Is the purpose of the repetition of the cross-validation to be able to calculate a test MSE which is completely independent of the training data?

Yes you are more or right in this case. Although strictly speaking, in CV there is no overlap between for every tested part used, it was never used in fitting the model. However for every fold train - test, although we used the same tuning parameter (n comps in this case), the principal components of the final model will slightly different. I.e the model used to test fold 1 will be differ from the model used to test fold 2. Also the test set is much smaller and might not reflect well the error in predicting unseen data.

So a better to circumvent this is to estimate the tuning parameter first using CV in your train set, fit a final model using the estimated parameter, then check this on the test set.

I think the motivation for doing it on the whole dataset first might be to show you the number of components choosen might not different that much.

Is the reason the lowest training MSEs for the first and second cross-validations different because less of the data was used in the second instance?

Yes this is part of the reason. The other reason is that the MSEs from the first part was done in round robbin style, i.e 10 different test-train combinations, whereas in the second half, it is strictly, 1 model on full train data, use 6 components, predict in test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.