I am trying to optimize the hyperparameters for a Gaussian process. I am using a squared exponential kernel, where I am optimizing three parameters.

$$k_y(x_p,x_q) = \sigma^2_f \exp\left(-\frac{1}{2l^2}(x_p-x_q)^2\right) + \sigma^2_n\delta_{pq}$$

As described by Rasmussen I am maximizing log marginal likelihood using its derivatives, but for some combination initial values of hyperparameters, I am occasionally getting negative values of $\sigma_f$ and $\sigma_n$. Are the negative values acceptable?

Many of the times people optimize the parameters in log domain to avoid negative values of hyperparameters, but I do not understand how ppl do that.

  • 3
    $\begingroup$ Clearly, based on the form of your kernel, a negative value of $\sigma_f$ and $\sigma_n$ is equivalent to their absolute value. $\endgroup$ – Bridgeburners Aug 23 '17 at 13:34

It's "acceptable" in the sense that, if you just take the absolute value, you'll get the same model out, so just do that and then it doesn't matter.

But optimizing in the log domain often works better for these kinds of parameters anyway. You do that just by reparameterizing your kernel as $$ k_y(x_p, x_q) = \exp(s_f) \exp\left( - \frac{1}{2 l^2} (x_p - x_q)^2 \right) + \exp(s_n) \delta_{pq} $$ and then optimizing $s_f$, $s_n$ instead of $\sigma_f$, $\sigma_n$. (You might want to do the same for $l$.)

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.