Gaussian process regression toy problem

I was trying to gain some intuition for Gaussian Process regression, so I made a simple 1D toy problem to try out. I took $x_i=\{1,2,3\}$ as the inputs, and $y_i=\{1,4,9\}$ as the responses. ('Inspired' from $y=x^2$)

For the regression I used a standard squared exponential kernel function:

$$k(x_p,x_q)=\sigma_f^2 \exp \left( - \frac{1}{2l^2} \left|x_p-x_q\right|^2 \right)$$

I assumed that there was noise with standard deviation $\sigma_n$, so that the covariance matrix became:

$$K_{pq} = k(x_p,x_q) + \sigma_n^2 \delta_{pq}$$

The hyperparameters $(\sigma_n,l,\sigma_f)$ were estimated by maximizing the log likelihood of the data. To make a prediction at a point $x_\star$, I found the mean and variance respectively by the following

$$\mu_{x_\star} = k_\star^T (\mathbf{K}+\sigma_n^2\mathbf{I})^{-1} y$$ $$\sigma_{x_\star}^2 = k(x_\star,x_\star)-k_\star^T(\mathbf{K}+\sigma_n^2\mathbf{I})^{-1} k_\star$$

where $k_\star$ is the vector of the covariance between $x_\star$ and inputs, and $y$ is a vector of the outputs.

My results for $1<x<3$ are shown below. The blue line is the mean and red lines mark the standard deviation intervals.

I'm not sure if this is right though; my inputs (marked by 'X's) do not lie on the blue line. Most examples I see have the mean intersecting the inputs. Is this a general feature to be expected?

• If I had to guess, in the examples you were looking at there was no residual error. In that case the line would pass through all the points. – guy Jul 10 '13 at 14:43
• @Guy exactly right. – user25658 Jul 10 '13 at 16:37

The mean function passing through the datapoints is usually an indication of over-fitting. Optimising the hyper-parameters by maximising the marginal likelihood will tend to favour very simple models unless there is enough data to justify something more complex. As you only have three datapoints, which are more or less in a line with little noise, the model that has been found seems pretty reasonable to me. Essentially the data can either be explained as a linear underlying function with moderate noise, or a moderately non-linear underlying function with little noise. The former is the simpler of the two hypotheses, and is favoured by "Occam's razor".

• Thanks for the input. Can you tell me more about "over-fitting"; is it a positive/negative feature? – Comp_Warrior Jul 10 '13 at 13:38
• over-fitting is a negative thing, it basically means the model is memorising the random variation in the data, which tends to make generalisation performance worse. Ideally you want the model to learn the underlying form of the data while ignoring the noise contaminating it. Most good machine learning textbooks, will cover this in an early chapter. – Dikran Marsupial Jul 10 '13 at 15:35
• just out of interest, why the downvote? – Dikran Marsupial Jul 10 '13 at 17:01
• I did not downvote you; in fact I upvoted! – Comp_Warrior Jul 10 '13 at 18:11
• no problem Comp_Warrior, I didn't think it was you, but someone did downvote my answer and I would be happy to have some feedback on why. We are all fallible and if I have got something wrong in my answer, I am keen to correct it. – Dikran Marsupial Jul 10 '13 at 18:33

You are using the Kriging estimators with the addition of a noise term (known as a nugget effect in the Gaussian process literature). If the noise term was set to zero, i.e.,

$$\sigma^2_n \delta_{pq}=0$$

then your predictions would act as an interpolation and pass through the sample data points.

This looks OK to me, in the GP book by Rasmussen it definitely shows examples where the mean function does not pass through each data point. Note that the regression line is an estimate for the underlying function, and we're assuming that the observations are the underlying function values plus some noise. If the regression line based through all three points it would essentially be saying that there is no noise in the observed values.

You could force a no noise assumption by setting $\sigma_n = 0$, and just optimizing the other hyper-parameters.

I also suspect that the hyper-parmeter $l$ is being set to a relatively large value, giving a very shallow function.

You could try holding $l$ fixed at various smaller values, and see how that changes the curve. Perhaps if you forced $l$ to be a bit smaller, the regression line would pass through all of the data points.

As noted by Dikran Marsupial, this is a built in feature of Gaussian Processes, the marginal likelihood penalizes models that are too specific, and prefers ones that can explain many data sets.