2
$\begingroup$

How to interpret big signal variance of optimized gaussian rbf kernel in gaussian process regression model? How to interpret big signal variance in general for this kernel? The function that is optimized is log-marginal likelihood: $$ log p(y|X) = {-\frac{1}{2}y^T(K+\sigma^2_n\mathcal{I})^{-1}y} {- \frac{1}{2}log|K + \sigma^2_n\mathcal{I}|} - \frac{n}{2}log2\pi $$

and the kernel function:

$$ k_{SE}(x_p, x_q) = \alpha^2 exp\big(-\frac{1}{2}(x_p - x_q)^T \Lambda^{-1} (x_p - x_q)\big) $$ where $\Lambda$ is a diagonal 'lengthscales' matrix.

example using GPflow

h(x) = a*x

[edit 1]: example using GPflow with different starting values for hyperparameters Here I just plot predictions of models. Models had set starting values of parameters according to labels in the image. 0 value means I did not set hyperparameters (should be default=1). different starting values

Is this because of points being very linear, so there all $ x_i $ from the dataset has the same "impact" on predicted $x_*$ (big lengthscales)? If so, how can one interpret big value of gaussian kernel variance?

$\endgroup$
1
$\begingroup$

In the first, example, it's picking a very long lengthscale and high variance, which is strange given that there's no noise in your data. You might be falling into a local minimum. Are you able to try different starting points for the hyperparameter optimization?

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I've updated question (2nd image had to be replaced due to lack of points). It looks like if I play with starting values of hyperparameters I end up with an oversimplified model. $\endgroup$ – gargne Apr 25 '17 at 11:10
  • $\begingroup$ Are you fitting the hyperparameters to maximize marginal likelihood? If so, do you have the final hyperparameters? What is a2? It looks like what is happening in the poorly behaving cases is that the posterior covariance between any point not in the training set and any point in the training set is 0, so it just predicts the prior (0-mean, some constant variance) for every non-training point. $\endgroup$ – Kevin Yang Apr 27 '17 at 4:26
  • $\begingroup$ a2 is $\alpha^2$, the signal variance or the scaling factor of a Gaussian kernel (I am probably calling it wrong). How can one interpret this big variance (kern.variance)? $\endgroup$ – gargne May 6 '17 at 19:02
  • $\begingroup$ Can you write out the equation for your Gaussian kernel? I'm still not sure which parameter you're concerned about. $\endgroup$ – Kevin Yang May 8 '17 at 19:02
  • $\begingroup$ I updated the question $\endgroup$ – gargne May 9 '17 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.