Cross validation and parameter optimization I have a question about the parameter optimization when I use the 10-fold cross validation.
I want to ask that whether the parameters should fix or not during every fold's model training , i.e. (1) select one set of optimized parameters for every fold's average accuracy.
or
(2) I should find the optimized parameter for every fold and then every fold uses different optimized parameters to train its model then test on the fold's test data respectively, and finally average every fold's accuracy as result?
Which one is the correct method for cross validation? Thanks a lot.
 A: I think the currently accepted answer is incomplete in an unfortunate way. I do not agree with the sentence 

The purpose of cross-validation is to identify learning parameters
  that generalise well across the population samples we learn from in
  each fold.

This is indeed one very important application of cross validation, but not the only one. Usually, you want to do two things:


*

*Build the best model you can

*Get an accurate impression of how well it performs


Now, to complete objective 1 depending on your algorithm you might need to tune some hyperparameters and this is indeed often done by cross validation. But this does not yet help you with objective 2. For this you need to basically nest the cross validation, like this:


*

*Seperate entire data into n folds

*For each, fold seperate the training data again into subfolds

*Use cross validation on the subfolds to learn good hyperparameters

*With these hyperparameter build a model on the training data of that fold

*Test the model on the test data

*Repeat on next fold


To build a good model you just need the inner cross validation. You will still need to do so to get a good model. But to get a good estimate of your model performance you need to perform the entire process of model building inside a cross validation scheme. This also includes steps like imputation, etc.
A: Let us firstly distinguish between two sets of parameters: model parameters (e.g. weights for features in regression), and parameters to the learning algorithm (and hyperparameters). The purpose of cross-validation is to identify learning parameters that generalise well across the population samples we learn from in each fold.
More specifically: We globally search over the space over learning parameters, but within each fold, we fix learning parameters and learn model parameters. The outcome should be learning parameters that produce on average the best performance in all folds. We can then use these to train a model on the entire dataset.
