I am confused with how the machine learning pipeline utilizes cross validation. Based on How should Feature Selection and Hyperparameter optimization be ordered in the machine learning pipeline??, I understand the purpose of cross validation to be estimating model stability and performance. Then once we are pleased with the results from cross validation (e.g. overall, we have selected the same features or hyperparameters), we can confidently run feature selection and hyperparameter tuning again on the full training data and use those results to build our final model.

A couple of questions, do we ever make direct decisions about our final model from cross validation results? What I mean by this is, once we have our top selected features or top performing hyperparameters from cross validation, is it valid to train models using those top features/model hyperparameters on the full training data set, then test results on the real test set?

What are the advantages/disadvantages of the first paragraph approach and the second paragraph approach?

Please list references.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.