2
$\begingroup$

How to interpret big signal variance of optimized gaussian rbf kernel in gaussian process regression model? How to interpret big signal variance in general for this kernel? The function that is optimized is log-marginal likelihood: $$ log p(y|X) = {-\frac{1}{2}y^T(K+\sigma^2_n\mathcal{I})^{-1}y} {- \frac{1}{2}log|K + \sigma^2_n\mathcal{I}|} - \frac{n}{2}log2\pi $$

and the kernel function:

$$ k_{SE}(x_p, x_q) = \alpha^2 exp\big(-\frac{1}{2}(x_p - x_q)^T \Lambda^{-1} (x_p - x_q)\big) $$ where $\Lambda$ is a diagonal 'lengthscales' matrix.

example using GPflow

h(x) = a*x

[edit 1]: example using GPflow with different starting values for hyperparameters Here I just plot predictions of models. Models had set starting values of parameters according to labels in the image. 0 value means I did not set hyperparameters (should be default=1). different starting values

Is this because of points being very linear, so there all $ x_i $ from the dataset has the same "impact" on predicted $x_*$ (big lengthscales)? If so, how can one interpret big value of gaussian kernel variance?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In the first, example, it's picking a very long lengthscale and high variance, which is strange given that there's no noise in your data. You might be falling into a local minimum. Are you able to try different starting points for the hyperparameter optimization?

Improve this answer
$\endgroup$
6
  • $\begingroup$ I've updated question (2nd image had to be replaced due to lack of points). It looks like if I play with starting values of hyperparameters I end up with an oversimplified model. $\endgroup$
    – gargne
    Apr 25, 2017 at 11:10
  • $\begingroup$ Are you fitting the hyperparameters to maximize marginal likelihood? If so, do you have the final hyperparameters? What is a2? It looks like what is happening in the poorly behaving cases is that the posterior covariance between any point not in the training set and any point in the training set is 0, so it just predicts the prior (0-mean, some constant variance) for every non-training point. $\endgroup$
    – Kevin Yang
    Apr 27, 2017 at 4:26
  • $\begingroup$ a2 is $\alpha^2$, the signal variance or the scaling factor of a Gaussian kernel (I am probably calling it wrong). How can one interpret this big variance (kern.variance)? $\endgroup$
    – gargne
    May 6, 2017 at 19:02
  • $\begingroup$ Can you write out the equation for your Gaussian kernel? I'm still not sure which parameter you're concerned about. $\endgroup$
    – Kevin Yang
    May 8, 2017 at 19:02
  • $\begingroup$ I updated the question $\endgroup$
    – gargne
    May 9, 2017 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.