overfit gaussian regression

Question

I have recently read a lot of posts about how maximizing the marginal likelihood in Gaussian Regression can cause overfitting. What is the best way out then? If we let the hyperparameters take values that maximize the chances of describing the data, then overfitting is a problem. But isnt that something that must be done?

Answer 1

There are numerous way how to avoid overfitting in general, let me mention some of them:

• To use the Cross-Validation, i.e. to take a part of the data to train the model and the rest to validate it.
• In the Bayesian context, the empirical Bayes seems to be a reasonable solution.

Both approaches have been already discussed here.

Answer 2

The problem with Gaussian processes regression is that Maximum Likelihood estimate for hyperparameters of covariance function can be bad: with this estimate we get true values as prediction for training points, bot we get constant value (equals mean) as prediction for other points.

I solved this problem using Bayesian regularization for hyperparameters: we impose a prior distribution on hyperparameters, and use MAP (Max of posterior distribution) as estimate. If you want to solve a specific problem - this approach will work for sure; if you try to build a general-purpose package - you have to be careful.