# How adding covariance noise in Gaussian processes to prevent overfitting?

I am told, in Gaussian Processes, adding covariance function noise to others, say SEiso or Materns, cause a better result, since it prevents from over fitting. I appreciate if someone could put more light on it !

-

The covariance essentially specifies how similar the regression function should be as a function of the attributes. The squared exponential function encodes the prior knowledge that the posterior should be a smooth function (on some characteristic length-scale governed by the hyper-parameters). If the function that generated the observations is not smooth however but has a random element (which will generally be the case), then maximising the marginal likelihood will end up encouraging the hyper-parameters to take on values that allow the GP to model these random variations, rather than concentrating on the larger scale smoothness.

Adding a noise term encodes the prior knowledge that the function should be fairly smooth (the SEiso bit) but that there will also be random variations superimposed on top. Then in model selection, vairations in the data can be explained as being due to either the difference in the input features (the SEiso bit) or just down to random meaningless variability in the data. There is then less pressure on the SEiso bit to explain the variability, and hence less pressure to make an SEiso component with short length-scales, which tend to generalise better.

Having said which, I have been performing some experiments with GP classifiers and learning the covariance function, and I found it didn't make a great deal of difference whether a noise term was used or not. The marginal likelihood is a statistic that is evaluated on a finite (normally quite small) sample of data, which means it has a non-negligible variance, which means that there is inevitably a danger of over-fitting if you perform model selection by maximising the marginal likelihood. That danger might be more substantial than the danger of model mis-specification (not including the right components in the covariance function).

-
Assume that you want to fit $\beta$ in the model $$[y_1, y_2] = [0,1]\beta + [\epsilon(s_1), \epsilon(s_1)],$$ where the noise $\epsilon$ are a zero mean Gaussian process with say Matérn covariance (any covariance function will do). So you have two observations at the same point but then the only difference between the observations are $\beta$, so $\beta = y_2 -y_1$ with probability 1.