Cross validation train and test error

Question

I came across this sort of flowchart:

Below the flowchart, this is what appears:

“Given a training set, cross-validation error is computed for each configuration of tuning parameters (λ,d). The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance. Using the best configuration of tuning parameters, we then train the models M2 and M3 on the original training set and use the original test set to compute the corresponding test RMSEs.”

• They are only mentioning the cross validation error (validation) and never mention the train cross validation error.
• Is the phrase “The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance” correct? I mean, assuming that by “lowest overall cross-validation error leads to the best model performance”, they are referring themselves to the “validation” error of the cross validation technique, I wonder why are they making such assumption? Should we care about the averaged train cross validation error or just the averaged validation error?

I am using a library to play with recommender systems which has a parameter called return_train_measures = True. Then it throws both, train and test errors:

Answer 1

The goal of cross-validation is to generate a more accurate performance estimation that just performing standard evaluation with a single test set, since this often leads to overfitting. Therefore, the sentence "The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance" should be correct for most cases, however, it can also happen that another model performs better on the test set. I think that the sentence just expresses the general assumption that is made when performing cross-validation.

The train error you are talking about is just used internally within the training of your model. You can take the training error into consideration when you want to identify overfitting (very low training error, but high validation error can indicate overfitting), but for the final model selection you should always use the validation error.