I came across this sort of flowchart:
Below the flowchart, this is what appears:
“Given a training set, cross-validation error is computed for each configuration of tuning parameters (λ,d). The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance. Using the best configuration of tuning parameters, we then train the models M2 and M3 on the original training set and use the original test set to compute the corresponding test RMSEs.”
- They are only mentioning the cross validation error (validation) and never mention the train cross validation error.
- Is the phrase “The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance” correct? I mean, assuming that by “lowest overall cross-validation error leads to the best model performance”, they are referring themselves to the “validation” error of the cross validation technique, I wonder why are they making such assumption? Should we care about the averaged train cross validation error or just the averaged validation error?
I am using a library to play with recommender systems which has a parameter called return_train_measures = True. Then it throws both, train and test errors: