4
$\begingroup$

I'm currently experimenting with Gaussian processes. I decided to use matlab + gpml (http://www.gaussianprocess.org/gpml/code/) for playing around with Gaussian processes a bit.

I'd like to do Gaussian process regression on 2d data (2d inputs and 1d output). For that I created some simple test data:

    #x y z
    0 0 -1
    0 7 -1
    3 7 -1
    8 3 -1
    5 5 -1
    8 8 5

The result looks like this (I used a unit lengthscale and magnitude with the squared exponential cov function):

x dimension fixed to 8

Now I wanted to add some variance at position (8,8), so I added the following to the training data:

    8 8 -1
    8 8 10
    8 8 -10
    8 8 -8
    ...

I expected the variance at this point to increase a lot. While the mean increased at this point, the variance hardly changed at all. This is the result: again x is fixed to 8. Data has added variance in (8,8)

I tried to play around with the hyperparameters, but I couldn't get a result that looked like the one I expected: the variance increasing at (8,8)

I'm really stuck here, so I'd really appreciated, if anyone could explain this behavior to.

Improve this question
$\endgroup$
5
  • $\begingroup$ @MånsT: Thanks for the edits. I was just about to make them myself and was happy to see you beat me to it. :) $\endgroup$
    – cardinal
    Jun 15, 2012 at 13:53
  • $\begingroup$ I don't think that I understand that behavior. Have you tried to do that in 1D? And by the way, have you tried to do the same, but introduce points not in $(8,8)$, but in $(8.1,8)$, $(8,7.9)$, etc.? $\endgroup$
    – Dmitry Laptev
    Jun 15, 2012 at 14:47
  • $\begingroup$ Did you generate your data from a Gaussian process? It doesn't appear so to me. Maybe that is partof your problem. $\endgroup$
    – Michael R. Chernick
    Jun 15, 2012 at 14:50
  • $\begingroup$ yes, the test data was not generated by a GP. but shouldn't it be possible to use GP regression on any distribution? $\endgroup$
    – Tobias Domhan
    Jun 16, 2012 at 14:27
  • $\begingroup$ So I'm assuming that this is just a mis-specification problem. I thought it wouldn't be a problem to model any kind of distribution with a GP, but in this case we got a function that has no noise at all places, expect one, where there is some huge noise. So the Gaussian prior doesn't fit the problem. (and also the SE cov function is not a great fit either) $\endgroup$
    – Tobias Domhan
    Jun 19, 2012 at 12:33

4 Answers 4

Reset to default
3
$\begingroup$

If you have different y-values for the point (8,8) then you are supposing that there is noise present. You should model this noise, for instance in the covariance function. Try a covariance like this one:

covfunc = {'covSum',{'covSEiso','covNoise'}};
Improve this answer
$\endgroup$
5
  • $\begingroup$ wouldn't that only model noise in the observations? that is, if you were not sure, which x and y a new observation will be associated with? $\endgroup$
    – Tobias Domhan
    Jun 16, 2012 at 8:46
  • 2
    $\begingroup$ +1 to Steven. @Tobias, modelling the noise in the observations is going to be important here as you have several different values for y for x = (8,8). covNoise basically helps you to model the conditional variance in y. $\endgroup$
    – Dikran Marsupial
    Jun 18, 2012 at 9:12
  • 1
    $\begingroup$ mhh could you explain that further? I mean, sure I have different values at that specific location, but shouldn't that be modeled by the variance at that point rather than with noise in the covariance function? the location where the observation are coming from is known exactly, it's (8,8), but the set of observations at this location just has a higher variance than in other locations. (or is the latter something that can't be modeled by a GP) (sorry if I'm asking too trivial questions, btw) $\endgroup$
    – Tobias Domhan
    Jun 18, 2012 at 13:15
  • 1
    $\begingroup$ Your covariance does not assume noise. Therefore, it assumes a GP that has zero predictive variance at each point. Check for instance Figure 2.2b from the GPML book for a nice illustration of this property. Now if your data has non-zero variance at some points then your initial assumption was not true. Basically your data is contradicting your assumption (covariance). $\endgroup$
    – Steven
    Jun 24, 2012 at 22:19
  • $\begingroup$ Not sure if that is true. The usual way (and only way I know) is to have f(x) be the mean of a Gaussian likelihood (likGauss in gpml), which has a parameter sn for the standard deviation. This way we can make the difference between the predictive mean f(x), and the observations y(x), which your model here with covfunc can't do. $\endgroup$
    – samlaf
    Mar 5, 2019 at 18:39
3
$\begingroup$

I hope it's ok if I provide an answer for my own question.

While I totally agree that this is a mis-specification problem and the data can't be modelled with a GP appropriately, I just realized that when initially asking the question I had a wrong understanding of how to interpret the variance. So the idea that I initially had, was that the variance given by the GP at a location is the variance of the data at that point. Which is not the case.

So there ware basically two things that are essential to understand here:

  1. the variance only depends on the location x and not on the target y. (see "Gaussian Processes for Machine Learning" page 35 formula 2.26)
  2. adding samples to the data set will always decrease but never increase the variance (or stay the same) (see page 31)

So all in all: the variance will never increase when adding data.

Note: the proof for that is left as an exercise in the GPML book (exercise 2.9.4), which I wasn't able to follow through, so if anyone did the proof, it'd be great if you could add it.

Improve this answer
$\endgroup$
2
$\begingroup$

It may be that you are plotting the uncertainty in the conditional mean (i.e. the regression) rather than the predictive distribution. If anything adding additional points for some value of x will decrease the variance of the regression (the uncertainty in the estimate of the conditional mean), but widen the predictive distribution (the uncertainty in where the observations might reasonably expect to be for some value of x).

Improve this answer
$\endgroup$
1
1
$\begingroup$

At first, including points with similar inputs can lead to ill-conditioned covariance matrix, especially if you specify a small noise term, so it is better to use some design in small region which includes your point.

Now let's go to the question. I think that some package for heteroskedastic regression is what you need. There are a couple of them, because the problem is intractable and one requires some approximate inference method to proceed gaussian processes regression in this case (there are many ways to proceed approximate inference). I mention two MATLAB packages:

You can try both or find some other implementations following links from these papers.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.