1
$\begingroup$

I just touched Gaussian processes two weeks ago. I am not very familiar with the selection of a model and its hyperparameters. Here is the demo code that I run for a 2-D Gaussian processes regression. Its output is not what I expected.

% produce the training set for regression.
% Here, the regression target Y is the sum of input.
X1train = linspace(-4.5,4.5,10);
X2train = linspace(-4.5,4.5,10);
X = [X1train' X2train'];
Y = sum(X,2);

% produce the test set for regression
Xtest = [1 2];

% set the hyperparameters
covfunc = {@covMaterniso, 3};
ell = 1/4; sf = 1;
hyp.cov = log([ell; sf]);
likfunc = @likGauss;
sn = 0.1;
hyp.lik = log(sn);

% implement regression
[ymu ys2 fmu fs2] = gp(hyp, @infExact, [], covfunc, likfunc, X, Y, Xtest);

The result is: ymu = 0.131695275851991, but what I expected is ymu = 1 + 2 = 3

$\endgroup$
  • $\begingroup$ It seems the issue is not related to the Matlab/GPML implementation, so the question should perhaps be retitled. $\endgroup$ – Juho Kokkala Jul 2 '14 at 10:30
3
$\begingroup$

This seems to be a modelling problem rather than an implementation problem. Let us plot the predictions (ymu) from your model and training data with more points:

%More testdata
[Xtest1, Xtest2] = meshgrid(-4.5:0.1:4.5, -4.5:0.1:4.5);
 Xtest = [Xtest1(:) Xtest2(:)];

% implement regression 
[ymu ys2 fmu fs2] = gp(hyp, @infExact, [], covfunc, likfunc, X, Y, Xtest);

pcolor(Xtest1,Xtest2, reshape(ymu,size(Xtest1)))
shading('flat')

Model prediction with length-scale parameter 0.25

The axes of the image represent the inputs Xtest1,Xtest2, and the color is the predicted mean. We see that the model output deviates from 0 only near the training points. This is due to the short length scale hyperparameter (ell) that causes the model to learn only short range phenomena. Let us try modifying that (all other code kept unchanged)

ell = 4

Model output with lengthscale parameter 4

Now the predicted output is much closer to your ground truth. Indeed, the prediction at $(1,2)$ is now $2.9$. It seems that (with the training data kept fixed), the longer the length-scale, the closer the prediction at your testpoint gets to 3.

Another observation is that all your training data are located at the $x_1=x_2$ line, so it is rather optimistic to hope that any model would learn what happens outside that line without making strong assumptions. Besides changing the hyperparameters, you could also play with adding training data at various other points, e.g., in a 2-dimensional grid.

$\endgroup$
  • $\begingroup$ Thank you very much. Your suggestions are very helpful for me to understand more about GP. And, I have a new question: Does GP have serious overfitting problem? If there are some outliers (e.g.10%), could GP automatically avoid their influence? $\endgroup$ – eirc Jul 3 '14 at 0:47
  • $\begingroup$ If my answer was helpful, please consider upvoting/accepting (though you may want to wait some time for other, perhaps better, answers before accepting.) The new question(s) you have would be better posted as a separate question (or two, if the overfitting question is unrelated to the outliers). These comments are meant for clarifications about the questions/answers, not for general discussion. $\endgroup$ – Juho Kokkala Jul 3 '14 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.