18
$\begingroup$

I have a question about the parameter optimization when I use the 10-fold cross validation.

I want to ask that whether the parameters should fix or not during every fold's model training , i.e. (1) select one set of optimized parameters for every fold's average accuracy.

or

(2) I should find the optimized parameter for every fold and then every fold uses different optimized parameters to train its model then test on the fold's test data respectively, and finally average every fold's accuracy as result?

Which one is the correct method for cross validation? Thanks a lot.

Improve this question
$\endgroup$
1
  • $\begingroup$ This question: stats.stackexchange.com/questions/1826/… has two great answers (highest score), i think they could help you in your question. The second is exactly what you want. OBS.: I'd like to write this as a comment, but I can't do it, so i've answered it. $\endgroup$
    – Augusto
    Commented Nov 10, 2012 at 22:30

2 Answers 2

Reset to default
15
$\begingroup$

Let us firstly distinguish between two sets of parameters: model parameters (e.g. weights for features in regression), and parameters to the learning algorithm (and hyperparameters). The purpose of cross-validation is to identify learning parameters that generalise well across the population samples we learn from in each fold.

More specifically: We globally search over the space over learning parameters, but within each fold, we fix learning parameters and learn model parameters. The outcome should be learning parameters that produce on average the best performance in all folds. We can then use these to train a model on the entire dataset.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Sorry, the kind of parameter I questioned is hyperparameters as you said. $\endgroup$
    – Kevin
    Commented Nov 8, 2012 at 16:12
  • $\begingroup$ Such as the parameters c and g in libSVM. So, I should use the same c and g to train every fold's model as the (2) method I mentioned above then pick the best one? Thank you very much. $\endgroup$
    – Kevin
    Commented Nov 8, 2012 at 16:20
  • 2
    $\begingroup$ The (1) and (2) above weren't clear to me so I didn't refer to them explicitly. You should search the space of c & g that maximises your task metric when averaged across the folds. So you set c & g to some value, run the training and testing procedure on all folds, average them, keep that score, modify c or g, and repeat. Ultimately, you find the best c & g, and you can train the final model on all your data. $\endgroup$
    – Joel
    Commented Nov 9, 2012 at 1:17
  • $\begingroup$ Thank you very much. I try to summarize the answers. The data was split to 10 fold: fold-1:(train1,test1)...fold-10:(train10,test10) Then use a (c1,g1) to train and test on fold-1 to fold-10, average all fold's accuracy. Try another (c2,g2) to do the same process...repeat until I find the best (c,g). And the best (c,g)'s average accuracy will be the result of my 10-fold cross validation. $\endgroup$
    – Kevin
    Commented Nov 9, 2012 at 5:31
  • 1
    $\begingroup$ Sounds correct... $\endgroup$
    – Joel
    Commented Nov 11, 2012 at 2:43
19
$\begingroup$

I think the currently accepted answer is incomplete in an unfortunate way. I do not agree with the sentence

The purpose of cross-validation is to identify learning parameters that generalise well across the population samples we learn from in each fold.

This is indeed one very important application of cross validation, but not the only one. Usually, you want to do two things:

  1. Build the best model you can
  2. Get an accurate impression of how well it performs

Now, to complete objective 1 depending on your algorithm you might need to tune some hyperparameters and this is indeed often done by cross validation. But this does not yet help you with objective 2. For this you need to basically nest the cross validation, like this:

  • Seperate entire data into n folds
  • For each, fold seperate the training data again into subfolds
  • Use cross validation on the subfolds to learn good hyperparameters
  • With these hyperparameter build a model on the training data of that fold
  • Test the model on the test data
  • Repeat on next fold

To build a good model you just need the inner cross validation. You will still need to do so to get a good model. But to get a good estimate of your model performance you need to perform the entire process of model building inside a cross validation scheme. This also includes steps like imputation, etc.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Erik, could you please provide a reference of how to do this process? $\endgroup$
    – chao
    Commented May 21, 2015 at 15:56
  • $\begingroup$ Hi @Erik , so would the general order of analyses be (1) Find optimal tuning parameters with cross-validation, (2) Retrain model (with the gained tuning parameters) on the whole training dataset to get the model parameters, and (3) See the overall performance estimate of this method by using nested cross-validation? What I am confused about is that different hyperparameters may be chosen in the process of nester CV, so the nested CV would not be specifically investigating the overall performance of the hyperparameter/model parameters we gained above? $\endgroup$
    – Michelle
    Commented Sep 2, 2017 at 10:40
  • 2
    $\begingroup$ I'm a bit late to the conversation, but I'd like to point out this method is also called "nested" or "double cross-validation" and here's a nice explanation by Tom Fearn and an example with code in the scikit-learn documentation $\endgroup$
    – MD004
    Commented Jul 11, 2019 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.