I am going through the excellent book "Introduction to Machine Learning with Python," and reading about cross-validation. I can understand how it makes a more efficient use of the data than a typical train-test split, but the book also contains the caveat:
It is important to keep in mind that cross-validation is not a way to build a model that can be applied to new data. Cross-validation does not return a model... multiple models are built internally, but the purpose of cross-validation is only to evaluate how well a given algorithm will generalize when trained on a specific dataset.
So if cross-validation doesn't produce a model, does that mean that after performing cross validation, I need to build a model in the typical method using a train/test set? If so that would imply that my cross-validation scores would typically be higher than my final model's scores, since cross-validation makes a more efficient use of the data.
Or is it held that after cross-validation, I can simply train my model against all data without any further test set? That would mean that I've never tested my actual model, so it sounds wrong, but perhaps cross-validation is a valid test since it uses every sample in both training and test? If so it implies that my cross-validation scores would typically be lower than my final model's scores, since only my final model would be trained against all samples.