-2
$\begingroup$

Can you please provide method to get final best model from cross validation.k-fold cross validation we have k models and accuracy estimate by average of k models accuracy.I need to know about how we select model from k-fold cross validation method.Is final model is the model with maximum accuracy.Can you please explain with one example

$\endgroup$
0
$\begingroup$

Well, I will provide only a text based example and try to describe, very briefly, some tips for selecting the best models from a K-folds cross validation.

The idea of cross validating is to tune a modeling methods parameters on data which have known property values (y values, class labels, etc). For example, in Principal Component Regression you want to find the best number of principal components that gives you the best model. The real question is 'what the best model?'

For regression that is typically the most accurate model. There are caveats in classification tasks. So naively, or perhaps correctly we could assume that the parameter which offered the lowest error is the best. However, cross validation makes a pretty big assumption. It assumes that all future data will look just like the training data which was used to cross validate. If the there are potential discrepancies in the behavior of your new data or 'test set' and the training data picking the model which has the lowest error from K-folds may not be the best. This is because the model that you would have selected would be over fit to the training data and not be able to represent the nuances which can appear in the future or test data.

This is why you will see RMSECV (root mean squared errors of cross validation) and RMSET (root mean squared errors of training) and others in papers. Error on the training or cross validation set is not guaranteed to represent future data. So instead we often trade bias toward the training set for variance. In the example of PCR this can be done by selecting the model which uses fewer principal components than the model which affords the lowest error. This will ensure that our model is not trained specifically to the cross validated data and is perhaps more robust to new effects we might see in future data, but still captures the effect we want to model. That sort of test could be assessed by setting aside some data for 'validation' purposes. Again though, this is data and model dependent. There are cases where you don't want to do this sort of trade, or potentially you want to do the opposite trade. That's part of the art of modeling.

There are some very strong posts about K-folds on this website, and many others on blogs, and discussions in scientific papers available. I think I covered the very basic ideas here so you can get started on reading some other resources with a few of the key concepts introduced.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.