# How to increase variance in Gaussian Process regression?

I'm currently experimenting with Gaussian processes. I decided to use matlab + gpml (http://www.gaussianprocess.org/gpml/code/) for playing around with Gaussian processes a bit.

I'd like to do Gaussian process regression on 2d data. For that I created some simple test data:

    #x y z
0 0 -1
0 7 -1
3 7 -1
8 3 -1
5 5 -1
8 8 5


The result looks like this (I used a unit lengthscale and magnitude with the squared exponential cov function):

Now I wanted to add some variance at position (8,8), so I added the following to the training data:

    8 8 -1
8 8 10
8 8 -10
8 8 -8
...


I expected the variance at this point to increase a lot. While the mean increased at this point, the variance hardly changed at all. This is the result:

I tried to play around with the hyperparameters, but I couldn't get a result that looked like the one I expected: the variance increasing at (8,8)

I'm really stuck here, so I'd really appreciated, if anyone could explain this behavior to.

• @MånsT: Thanks for the edits. I was just about to make them myself and was happy to see you beat me to it. :) – cardinal Jun 15 '12 at 13:53
• I don't think that I understand that behavior. Have you tried to do that in 1D? And by the way, have you tried to do the same, but introduce points not in $(8,8)$, but in $(8.1,8)$, $(8,7.9)$, etc.? – Dmitry Laptev Jun 15 '12 at 14:47
• Did you generate your data from a Gaussian process? It doesn't appear so to me. Maybe that is partof your problem. – Michael Chernick Jun 15 '12 at 14:50
• yes, the test data was not generated by a GP. but shouldn't it be possible to use GP regression on any distribution? – Tobias Domhan Jun 16 '12 at 14:27
• So I'm assuming that this is just a mis-specification problem. I thought it wouldn't be a problem to model any kind of distribution with a GP, but in this case we got a function that has no noise at all places, expect one, where there is some huge noise. So the Gaussian prior doesn't fit the problem. (and also the SE cov function is not a great fit either) – Tobias Domhan Jun 19 '12 at 12:33

It may be that you are plotting the uncertainty in the conditional mean (i.e. the regression) rather than the predictive distribution. If anything adding additional points for some value of x will decrease the variance of the regression (the uncertainty in the estimate of the conditional mean), but widen the predictive distribution (the uncertainty in where the observations might reasonably expect to be for some value of x).

If you have different y-values for the point (8,8) then you are supposing that there is noise present. You should model this noise, for instance in the covariance function. Try a covariance like this one:

covfunc = {'covSum',{'covSEiso','covNoise'}};

• wouldn't that only model noise in the observations? that is, if you were not sure, which x and y a new observation will be associated with? – Tobias Domhan Jun 16 '12 at 8:46
• +1 to Steven. @Tobias, modelling the noise in the observations is going to be important here as you have several different values for y for x = (8,8). covNoise basically helps you to model the conditional variance in y. – Dikran Marsupial Jun 18 '12 at 9:12
• mhh could you explain that further? I mean, sure I have different values at that specific location, but shouldn't that be modeled by the variance at that point rather than with noise in the covariance function? the location where the observation are coming from is known exactly, it's (8,8), but the set of observations at this location just has a higher variance than in other locations. (or is the latter something that can't be modeled by a GP) (sorry if I'm asking too trivial questions, btw) – Tobias Domhan Jun 18 '12 at 13:15
• Your covariance does not assume noise. Therefore, it assumes a GP that has zero predictive variance at each point. Check for instance Figure 2.2b from the GPML book for a nice illustration of this property. Now if your data has non-zero variance at some points then your initial assumption was not true. Basically your data is contradicting your assumption (covariance). – Steven Jun 24 '12 at 22:19
• Not sure if that is true. The usual way (and only way I know) is to have f(x) be the mean of a Gaussian likelihood (likGauss in gpml), which has a parameter sn for the standard deviation. This way we can make the difference between the predictive mean f(x), and the observations y(x), which your model here with covfunc can't do. – samlaf Mar 5 at 18:39

At first, including points with similar inputs can lead to ill-conditioned covariance matrix, especially if you specify a small noise term, so it is better to use some design in small region which includes your point.

Now let's go to the question. I think that some package for heteroskedastic regression is what you need. There are a couple of them, because the problem is intractable and one requires some approximate inference method to proceed gaussian processes regression in this case (there are many ways to proceed approximate inference). I mention two MATLAB packages:

You can try both or find some other implementations following links from these papers.

I hope it's ok if I provide an answer for my own question.

While I totally agree that this is a mis-specification problem and the data can't be modelled with a GP appropriately, I just realized that when initially asking the question I had a wrong understanding of how to interpret the variance. So the idea that I initially had, was that the variance given by the GP at a location is the variance of the data at that point. Which is not the case.

So there ware basically two things that are essential to understand here:

1. the variance only depends on the location x and not on the target y
2. adding samples to the data set will always decrease but never increase the variance (or stay the same) (see "Gaussian Processes for Machine Learning" page 31)

So all in all: the variance will never increase when adding data.

Note: the proof for that is left as an exercise in the GPML book (exercise 2.9.4), which I wasn't able to follow through, so if anyone did the proof, it'd be great if you could add it.