Why is MSE used in cross validation when selecting optimum number of variables in model? I'm currently looking through An Introduction to Statistical Learning by Gareth James, more specfically Chapter 6. It discusses ways to select the optimal number of variables in a model using methods such as forward stepwise selection. In the lab, they use MSE in cross validation calculations to compare different sized models (e.g 3 predictors vs 4). In previous sections MSE has also been used in cross-validation settings. But in the case of selecting an optimal number of ceofficients, surely cross-validation is not appropriate here, as it uses MSE, which reduces as the number of predictors increases?
 A: When you do cross-validation, you check your model on data that the model did not see during the training process. By doing so, you penalize the model for picking up on mere coincidences in the training data, the most extreme example of which is memorizing the training data (connect the dots).
You are correct that more parameters should mean a better fit, but that’s on training data. When you go to unseen data in cross-validation, the expectation (maybe more like hope) is that there will be an improvement in fit on unseen data up to a point, and then the additional parameters will cause the model to overfit. That sweet spot right before the overfitting begins is the parameter count you would use.
There are subtleties in developing a good machine learning model, but that is the gist and why your book uses cross validation to find the optimal number of parameters. It is finding the sweet spot in performance on the unseen data.
A: crossvalidation uses "test set" MSE which does not necessarily reduce as number of predictors increases (whereas your statement would be true for training set MSE)
