2
$\begingroup$

When we do K-fold cross validation, we are testing how well our model is able to get trained by some data and then predict data it hasn't seen.

I selected 9 fold for training, and 1 fold for validation. Training set would be 8 images, and validation set would be 2 images. I have Trained my model with training set, and computed performance with validation set. I have 10 training sets, 10 validation sets, 10 models, and 10 errors.

now What can I do

Do I need to choose a predictive model after I did k-fold cross-validation?

I have read a lot. But I do not understand what is the next step

Improve this question
$\endgroup$
1
  • $\begingroup$ cross validation tells you what is the mean error and its variance when using your chosen model and hyperparameters. It is considered to be a good estimate of how this model performs on unseen data. $\endgroup$
    – Jan Kukacka
    Jul 8, 2019 at 10:07

2 Answers 2

Reset to default
1
$\begingroup$

The main goals behind K-fold cross validation are

  • Selecting one model among many with an objective criterai that relates to the model's usefulness.
  • Have a first idea on how the model will perform.

So, what I would do is to take the model than perforemd best in k-fold cross validation (or maybe, the simplest model that did well enough). Then you can train this model with your entire training dataset and start the testing phase (to further check for overfitting or other issues)

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ This answer is very ambiguous (in particular with the difficulties in understanding expressed in the question): you do not select one of the $k$ so-called surrogate models calculated during the cross validation. This answer refers to cross validating $n$ different models with e.g. $n$ different sets of hyperparameters. So inside all the cross validations, $n \times k$ surrogate models are trained. The $k$ surrogate models with the same hyperparameter set are assumed to be equal (or equivalent), and no selection takes place among them! $\endgroup$
    – cbeleites unhappy with SX
    Jul 10, 2019 at 10:59
1
$\begingroup$

Although it seems very late to put a comment for an answer to your question, I do not really agree with the answer provided above. If you run a random forest prediction model with a 10-fold-cross-validation, you get only one model and get the model performance with average statistics(usually). If you have two random forests with different model parameters, then you get two models and you run 10-fold-cross-validation two times.

Then you compare the model performances against each other. You can pick the best one out of the two models and train the model with the whole data and then run on the test data to get the final model performance. This is how I understand.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ And there is no particular reason to stop at two random forests with different model hyperparameters (e.g. number of trees, fraction of data for each tree, maximum number of features for each tree): you can try more different model hyperparameters to see which seems to perform best or well enough in cross-validation. $\endgroup$
    – Henry
    Oct 5 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.