Log marginal likelihood for Gaussian Process as per Rasmussen's Gaussian Processes for Machine Learning equation 2.30 is:

$$\log p(y|X) = -\frac{1}{2}y^T(K+\sigma^2_n I)^{-1}y - \frac{1}{2}\log|K+\sigma^2_n I|-\frac{n}{2}\log2\pi$$

Implementations typically apply gradient descent by default to maximize this over some family of $K$, where that is typically a sum/product of constant kernels, RBF kernels, periodic kernels, etc

It is not clear how convex this optimization problem is, in general. If we sample some data $X$, $y$ from a 'nice' process (i.e. preventing crazily-contrived examples of non-convexity that would never occur in real life data), is this problem convex for common kernels? (sums/products of constant, RBF, exponential sine-squared, etc)

  • 1
    $\begingroup$ To get started, see "Covariance estimation for Gaussian variables" beginning on p. 155 and exercise 7.4 in "Convex Optimization",Boyd and Vandenberghe, web.stanford.edu/~boyd/cvxbook $\endgroup$ – Mark L. Stone Apr 23 '18 at 23:02

Unfortunately not. We often have to use multiple restarts if we are using convex optimisation and even then you often find poor hyper parameters.

One direction people have gone done is to use Monte Carlo based optimisation and this seems to work ok but the cost of evaluating the log likelihood / it's gradient isn't exactly cheep.

There was a bit of a push to use Bayesian optimisation to solve this and the die hard GP crowd are a big fan of being fully Bayesian using Bayesian quadrature to marginalise over hyperparameters. However these both essentially aim to use GPs to solve the problems of GPs and there is sense of inception going on!

| cite | improve this answer | |
  • $\begingroup$ "to use GPs to solve the problems of GPs and there is sense of inception going on"... I'm interested in 'model stacking' techniques, and this sounds interesting. Do you recommend any references illustrating this inception you speak of? $\endgroup$ – Scott Apr 22 '19 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.