When MSE for CV is greater than test MSE? In an introduction to statistical learning book, on page 311 there is a figure which compares MSE vs the number of leaves.

I like to know the reason that Cross-Validation's Mean Square Error curve is above test error. 
 A: On p. 310 they explain in detail what was done:


*

*the data set has a total of 263 cases.

*They split into training (132 cases) and test (131 cases) data sets,

*and further split the training set with 6-fold cross validation. 


So:


*

*Test set tests a model trained with 132 cases. 

*Cross validation tests surrogate models trained with 132 * 5/6 = 110 cases.


Cross validation surrogate models will be worse on average if the learning curve still increases from 110 to 132 cases (pessimistic bias of cross validation, well known effect). 
Side notes:


*

*If a final model is built using all 263 cases for training, that may be still better -  but we don't have any independent data left to check that...

*Cross validation error looking better than test set error is usually a symptom of the splitting procedure not achieving independence*.  However, the random splitting procedure here is exactly the same for splitting the whole data set into train + test and to further split the train for cross validation. Such an optimistic bias due to incorrect splitting would therefore affect both test and cross validation errors. I.e. we cannot detect it with this setup.


* This can be intentional: most commonly if the cross validation is used for model selection/hyperparameter optimization instead of validation/verification of predictive performance.
A: Usually you would get worse results on your test set than you would get on your training / CV set. However, sometimes you can get worse results on your CV set, for ex. if you had a very small sample or you used very few folds in your CV procedure (so large chunks left out in each fold), since you would be leaving out a lot of data in each fold step, thus learning less and being able to discriminate less with each model (as opposed to the test set results where you would use all the training data to build your model).
