Cross-validation and building a final model when using hyperparameter optimization

Question

I am trying to build a Gaussian process (GP) regression for a problem in which each experiment is computationally expensive, using cross-validation. Here is how I do it:

• Build the GP regressor on the full available dataset, with hyperparameter optimization (anisotropic Gaussian kernel)
• Perform 10-fold cross validation using the optimized hyperparameter set from the previous step

Now, what model should I select as the output of my procedure?

1. The model trained on the full dataset, considering that its global performance is validated by each cross validation fold?
2. A compound of each of the 10 models from cross validation?
3. The model from cross validation with the highest score?

I'm currently going for #1 but I was opposed that my model was then not properly validated. But I think it is implicitly validated because I used the same hyperparameters in cross validation as in the model. #2 would perhaps be better but this does not feel right to me. #3, in my opinion, is not an option, because that could mean selecting a model which performs well just because the few validation cases are adapted to it.

Am I doing the process right?

Answer 1

No, you are not doing it right.

As a general principle, cross-validation does NOT estimate the performance of one particular model. Instead, it estimates the performance of a given model-building procedure. This is very important to understand when using cross-validation.

Your model-building procedure involves hyperparameter optimization. So when you do your cross-validation, you should use this exact procedure on each training set, i.e. you should perform hyperparameter optimization on each training set separately. The cross-validation will then give you a performance estimate. Afterwards, if you need a "final" model, you apply your model-building procedure to the full dataset, i.e. you perform hyperparameter optimization on the full data.

This should answer your questions ##1-3.

Note that your hyperparameter optimization is an empirical Bayes procedure that is conceptually similar to cross-validation: see Cross-validation vs empirical Bayes for estimating hyperparameters. If your model-building procedure used cross-validation instead of hyper-parameter optimization, then the logic described above would lead you to the so called nested cross-validation.

Answer 2

No, your performance estimator is likely to be optimistically biased because the test data in each fold of the cross-validation procedure was used to tune the hyper-parameters, and so to some extent the hyper-parameter setting can exploit the noise in the test samples. See my paper:

Gavin C. Cawley, Nicola L. C. Talbot, "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation", journal of Machine Learning Research, 11(Jul):2079−2107, 2010. (http://www.jmlr.org/papers/v11/cawley10a.html)

The procedure you have adopted (very common in machine learning) is discussed in section 5.3 "Another Example of Biased Evaluation Methodology".

To get an unbiased performance estimate from cross-validation, you need to cross-validate all steps involved in fitting the model, including hyper-parameter tuning and perform them independently in each fold of the cross-validation (this often requires nested cross-validation).

The final model is created by performing these steps using the full dataset. The key is to think of cross-validation as a method of estimating the performance of a procedure for fitting a model, rather than of the model itself.

Note that the marginal likelihood (or evidence for the model) is a statistic evaluated over a finite set of samples, so it will also have a finite variance, which means that optimising hyper-parameters by marginal likelihood maximisation can still be just as susceptible to over-fitting the model selection criteria as optimising cross-validation based performance estimates. Note in my paper I use GP classification as one of the examples. Marginal likelihood maximisation is not really a very Bayesian approach, marginalisation over the hyper-parameters would be more elegant. My maxim is "optimisation is the root of all evil in statistics", but unfortunately marginalisation is generally difficult or computationally expensive, or both!