How to increase variance in Gaussian Process regression? I'm currently experimenting with Gaussian processes.
I decided to use matlab + gpml (http://www.gaussianprocess.org/gpml/code/) for playing around with Gaussian processes  a bit.
I'd like to do Gaussian process regression on 2d data (2d inputs and 1d output). For that I created some simple test data:
    #x y z
    0 0 -1
    0 7 -1
    3 7 -1
    8 3 -1
    5 5 -1
    8 8 5

The result looks like this (I used a unit lengthscale and magnitude with the squared exponential cov function):

Now I wanted to add some variance at position (8,8), so I added the following to the training data:
    8 8 -1
    8 8 10
    8 8 -10
    8 8 -8
    ...

I expected the variance at this point to increase a lot. While the mean increased at this point, the variance hardly changed at all. This is the result:

I tried to play around with the hyperparameters, but I couldn't get a result that looked like the one I expected: the variance increasing at (8,8)
I'm really stuck here, so I'd really appreciated, if anyone could explain this behavior to.
 A: If you have different y-values for the point (8,8) then you are supposing that there is noise present. You should model this noise, for instance in the covariance function. Try a covariance like this one:
covfunc = {'covSum',{'covSEiso','covNoise'}};

A: I hope it's ok if I provide an answer for my own question.
While I totally agree that this is a mis-specification problem and the data can't be modelled with a GP appropriately, I just realized that when initially asking the question I had a wrong understanding of how to interpret the variance. So the idea that I initially had, was that the variance given by the GP at a location is the variance of the data at that point. Which is not the case.
So there ware basically two things that are essential to understand here:

*

*the variance only depends on the location x and not on the target y. (see "Gaussian Processes for Machine Learning" page 35 formula 2.26)

*adding samples to the data set will always decrease but never increase the variance (or stay the same) (see page 31)

So all in all: the variance will never increase when adding data.
Note: the proof for that is left as an exercise in the GPML book (exercise 2.9.4), which I wasn't able to follow through, so if anyone did the proof, it'd be great if you could add it.
A: It may be that you are plotting the uncertainty in the conditional mean (i.e. the regression) rather than the predictive distribution.  If anything adding additional points for some value of x will decrease the variance of the regression (the uncertainty in the estimate of the conditional mean), but widen the predictive distribution (the uncertainty in where the observations might reasonably expect to be for some value of x).
A: At first, including points with similar inputs can lead to ill-conditioned covariance matrix, especially if you specify a small noise term, so it is better to use some design in small region which includes your point.
Now let's go to the question. I think that some package for heteroskedastic regression is what you need. There are a couple of them, because the problem is intractable and one requires some approximate inference method to proceed gaussian processes regression in this case (there are many ways to proceed approximate inference). I mention two MATLAB packages:


*

*Variational lower bound hetroskedastic regression

*MAP Heteroscedastic regression
You can try both or find some other implementations following links from these papers.
