13
$\begingroup$

When performing k-fold cross validation, I understand that you obtain the accuracy metrics by pointing all the folds except one at that one fold and make predictions, and then repeat this process $k$ times. You can then run accuracy metrics on all of your instances (precision, recall, % classified correctly), which ought to be the same as if you calculated them each time and then averaged the result (correct me if I'm wrong).

The end result you want is a final model.

Do you average the models obtained to make your set of $k$ predictions to end up with the model that has the accuracy metrics obtained by the above method?

$\endgroup$
15
$\begingroup$

The aim of $k$-fold cross validation is not to produce a model; it is to compare models.

The results of a cross validation experiment could tell you that Support Vector Machines out-perform Naive Bayes on your data, or that the classifier's hyper parameters should be set to c for this particular data set. Armed with this knowledge, you then train a "production" classifier with ALL of the available data and apply it to your problem.

In many cases, it's not even clear how you would go about averaging several models. For example, what is the average of three decision trees or nearest neighbor classifiers?

It's important to keep in mind that the cross validation results are estimates, not guarantees, and these estimates are more valid if the production classifier is trained with a similar quality (and quantity) of data. There has been a fair amount of work on developing ways to use these estimates to perform inference; that is, to say, in a statistically sound way, that method A is generally superior to method B on these data.

$\endgroup$
  • 2
    $\begingroup$ What's a good reference on using k-fold cross validation estimates for inference? I'd love to read up on that if you have a good share. $\endgroup$ – tentaclenorm May 2 '14 at 3:06
  • 1
    $\begingroup$ A decent place to start might be iro.umontreal.ca/~lisa/bib/pub_subject/finance/pointeurs/… but there are a bunch of different approaches. $\endgroup$ – Matt Krause May 28 '14 at 16:21
  • 1
    $\begingroup$ one more thing to clarify: when we train "production" classifier using all data how we understand when to stop ? $\endgroup$ – Anton Dec 8 '18 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.