# Gaussian Process Regression/ Classification

How do we estimate parameters of the model while performing Gaussian Process Regression or Classification? While performing regression, we estimate parameters such that the model is the best fit to the data. Thus, we try to increase the likelihood of the model, given the data. However, how do we estimate the parameters of a Gaussian Process regression process, where we know that the posterior will pass through all observed data points? In that case, every model passes through all observed data points. How do we differentiate between models?

• Unless the covariance function of the GP has been very badly specified, the output of the GO will not pass through all of the observed data points. See the GPML book for a good guide on the use of GPs gaussianprocess.org/gpml Commented Oct 11, 2014 at 13:41
• Thanks! This is what is a little confusing. I have been following the book. The book says that we chose the prior based on our "belief" about the data. Then the posterior enables us to reject the functions that do not agree with the data. The original question has a figure about what I am trying to say. Once we observe the data, wont the functions sampled from the gaussian process look like it is in the figure? Commented Oct 11, 2014 at 17:59
• I think this is because the covariance function used is very flexible, if you performed the same exercise but with a linear covariance function the regression would no longer pass through the datapoints. Alternatively you could change the width of the basis functions to make them much larger and the same thing would probably happen. If you have MATLAB (or octave) then experimenting with the toolbox is a good way of getting an intuitive understanding. Commented Oct 11, 2014 at 18:22
• Okay. I think I am getting the picture. I will experiment with the toolbox and get back! Thanks! Commented Oct 11, 2014 at 18:43
• I played the code and realised where my assumptions were wrong. But then there is another fundamental issue. Although the output is no longer passing through the observed data points, the system is totally confident about them. I would appreciate it if you check this out stats.stackexchange.com/questions/119745/… Commented Oct 12, 2014 at 7:44

There are cases in which we observe noise free data, such as Bayesian quadrature or Bayesian optimisation.

Two options that can be used to tune the (hyper-)parameters are:

Marginal Likelihood: We may tune the (hyper-)parameters by maximising the log marginal likelihood:

$$\log P(y|x,\theta) = -0.5y^TK^{-1}y - 0.5\log|K| - c$$

Where $c$ is a constant, the first term is the data fit term and the second is the complexity penalty. As long as you are not adding $\sigma I$ (you will usually see $K^+ = K + \sigma I$) to your covariance matrix there will be a range of values of (hyper-)parameters that will interpolate your data. They vary by the rate they drop back to the mean function.

Warning: noiseless data can cause numerical issues in the inversion of $K$, so a little bit of 'jitter' may be unavoidable.

Leave One Out (LOO): You can also use LOO, simply work out the log predictive probability for each of your data points and find the (hyper-)parameters of your kernel to maximise this.

$$LOO(\theta | x,y) = \sum_{i=0}^n \log P(y_i|x,y_{-i},\theta)$$

Both of these methods are outlined in GPML Chapter 5.