3
$\begingroup$

Log marginal likelihood for Gaussian Process as per Rasmussen's Gaussian Processes for Machine Learning equation 2.30 is:

$$\log p(y|X) = -\frac{1}{2}y^T(K+\sigma^2_n I)^{-1}y - \frac{1}{2}\log|K+\sigma^2_n I|-\frac{n}{2}\log2\pi$$

Implementations typically apply gradient descent by default to maximize this over some family of $K$, where that is typically a sum/product of constant kernels, RBF kernels, periodic kernels, etc

It is not clear how convex this optimization problem is, in general. If we sample some data $X$, $y$ from a 'nice' process (i.e. preventing crazily-contrived examples of non-convexity that would never occur in real life data), is this problem convex for common kernels? (sums/products of constant, RBF, exponential sine-squared, etc)

$\endgroup$
1
  • 1
    $\begingroup$ To get started, see "Covariance estimation for Gaussian variables" beginning on p. 155 and exercise 7.4 in "Convex Optimization",Boyd and Vandenberghe, web.stanford.edu/~boyd/cvxbook $\endgroup$ Apr 23, 2018 at 23:02

1 Answer 1

5
$\begingroup$

Unfortunately not. We often have to use multiple restarts if we are using convex optimisation and even then you often find poor hyper parameters.

One direction people have gone done is to use Monte Carlo based optimisation and this seems to work ok but the cost of evaluating the log likelihood / it's gradient isn't exactly cheep.

There was a bit of a push to use Bayesian optimisation to solve this and the die hard GP crowd are a big fan of being fully Bayesian using Bayesian quadrature to marginalise over hyperparameters. However these both essentially aim to use GPs to solve the problems of GPs and there is sense of inception going on!

$\endgroup$
1
  • $\begingroup$ "to use GPs to solve the problems of GPs and there is sense of inception going on"... I'm interested in 'model stacking' techniques, and this sounds interesting. Do you recommend any references illustrating this inception you speak of? $\endgroup$
    – Him
    Apr 22, 2019 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.