# Cross-validation and building a final model when using hyperparameter optimization

I am trying to build a Gaussian process (GP) regression for a problem in which each experiment is computationally expensive, using cross-validation. Here is how I do it:

• Build the GP regressor on the full available dataset, with hyperparameter optimization (anisotropic Gaussian kernel)
• Perform 10-fold cross validation using the optimized hyperparameter set from the previous step

Now, what model should I select as the output of my procedure?

1. The model trained on the full dataset, considering that its global performance is validated by each cross validation fold?
2. A compound of each of the 10 models from cross validation?
3. The model from cross validation with the highest score?

I'm currently going for #1 but I was opposed that my model was then not properly validated. But I think it is implicitly validated because I used the same hyperparameters in cross validation as in the model. #2 would perhaps be better but this does not feel right to me. #3, in my opinion, is not an option, because that could mean selecting a model which performs well just because the few validation cases are adapted to it.

Am I doing the process right?

• Salut Romain, can you elaborate what you do in case 1. please. What do you mean train the model on the full dataset? You say "The model trained on the full dataset, considering that its global performance is validated by each cross validation fold?" and then also "I'm currently going for #1 but I was opposed that my model was then not properly validated". I am a bit confused. – Tom Dec 4 '18 at 10:48
• Hi, I'll try to be more specific. Say we have a 50 cases dataset. I fit the GPR on those 50 cases, and get an optimized set of hyperparameters + a GPR model. Then, I fit 10 cross-validation models (45/5 split) using these exact hyperparameters. I get a validation score for each of the 10 cross-validation models (that's the point). But is this enough to assume that the first model (the one using all the 50 cases for training) is validated? It seems to me that if I'm using the same set of hyperparameters, then yes. I'm looking for adversarial or support opinions about that. Hope this clarifies.. – Romain Reboulleau Dec 4 '18 at 13:04
• This is quite wrong. There are multiple threads here on nested cross-validation, take a look. – amoeba Dec 6 '18 at 12:47
• @amoeba Thank you for your comment. I had already read a few posts about nested cross-validation. What I've been doing is not nested cross-validation, as there is no hyperparameter optimization in the CV folds. Then, are you suggesting I should switch to nested CV? Why is hyperparameter optimization through maximum likelihood not good enough here? It seems to me that I'm only performing standard cross-validation (with fixed hyperparameters that happen to be optimized through maximum likelihood). – Romain Reboulleau Dec 6 '18 at 16:45
• I perform gradient otpimization of hyperparameters (from a few starting points), trying to maximize the likelihood (actually log-marginal likelihood, in python's sklearn.gaussian_process.GaussianProcessRegressor) of the hyperparameters, given the "training" data. Each hyperparameter value is also bounded by "reasonable" values to prevent irrelevant solutions. – Romain Reboulleau Dec 7 '18 at 9:10

No, you are not doing it right.

As a general principle, cross-validation does NOT estimate the performance of one particular model. Instead, it estimates the performance of a given model-building procedure. This is very important to understand when using cross-validation.

Your model-building procedure involves hyperparameter optimization. So when you do your cross-validation, you should use this exact procedure on each training set, i.e. you should perform hyperparameter optimization on each training set separately. The cross-validation will then give you a performance estimate. Afterwards, if you need a "final" model, you apply your model-building procedure to the full dataset, i.e. you perform hyperparameter optimization on the full data.

Note that your hyperparameter optimization is an empirical Bayes procedure that is conceptually similar to cross-validation: see Cross-validation vs empirical Bayes for estimating hyperparameters. If your model-building procedure used cross-validation instead of hyper-parameter optimization, then the logic described above would lead you to the so called nested cross-validation.

• +1. Sometimes I like to think of a model-building procedure as a ranking process. I rank possible models to find the best one. I care for the order rather than the magnitude of the performance index. – usεr11852 Dec 13 '18 at 0:07
• Thank you for this very clear answer. If I understand well: my cross-validation estimation of my model performance is slightly biaised (so may be too optimistic regarding the user criterion), but in the end, the "best" model is the one from option #1. I will also have a look at nested cross-validation to get a "benchmark" method of the hyper-parameter optimization. – Romain Reboulleau Dec 13 '18 at 11:16
• @RomainReboulleau "my cross-validation estimation of my model performance is slightly biaised (so may be too optimistic regarding the user criterion), but in the end, the "best" model is the one from option #1" -- yes, that's correct. – amoeba Dec 13 '18 at 11:48

No, your performance estimator is likely to be optimistically biased because the test data in each fold of the cross-validation procedure was used to tune the hyper-parameters, and so to some extent the hyper-parameter setting can exploit the noise in the test samples. See my paper:

Gavin C. Cawley, Nicola L. C. Talbot, "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation", journal of Machine Learning Research, 11(Jul):2079−2107, 2010. (http://www.jmlr.org/papers/v11/cawley10a.html)

The procedure you have adopted (very common in machine learning) is discussed in section 5.3 "Another Example of Biased Evaluation Methodology".

To get an unbiased performance estimate from cross-validation, you need to cross-validate all steps involved in fitting the model, including hyper-parameter tuning and perform them independently in each fold of the cross-validation (this often requires nested cross-validation).

The final model is created by performing these steps using the full dataset. The key is to think of cross-validation as a method of estimating the performance of a procedure for fitting a model, rather than of the model itself.

Note that the marginal likelihood (or evidence for the model) is a statistic evaluated over a finite set of samples, so it will also have a finite variance, which means that optimising hyper-parameters by marginal likelihood maximisation can still be just as susceptible to over-fitting the model selection criteria as optimising cross-validation based performance estimates. Note in my paper I use GP classification as one of the examples. Marginal likelihood maximisation is not really a very Bayesian approach, marginalisation over the hyper-parameters would be more elegant. My maxim is "optimisation is the root of all evil in statistics", but unfortunately marginalisation is generally difficult or computationally expensive, or both!

• Thank you, I did not think of cross-validation the way you explain it: this seems to be quite important. I will definitely have a look at your paper to improve my methodological knowledge. – Romain Reboulleau Dec 13 '18 at 11:19