Calibrating a Gaussian Process

Question

In Thoughts on Massively Scalable Gaussian Processes (or any other introduction to Gaussian Processes), authors claim that calibrating a Gaussian process is just maximizing:

$$\log(p|y,\theta)\propto-\frac{1}{2}[y^T(K_{\theta}+\sigma^2I)^{-1}y + \log|K_{\theta}+\sigma^2I|]$$

Where the optimization takes place over $\theta$. Does it mean that the kernel can be chosen by the algorithm itself ?

If I just plug a list of kernels (linear, polynomial, exponential) and select the best fit over their parameters, I am selecting the best model ? By best I mean the model from which I can expect the highest out of sample performance.

This way of thinking is very far from what is done when calibrating SVMs, where all the hyperparameters of the kernel (and the kernel itself) are selected by cross-validation.

Answer 1

Yes, for Gaussian Process models Maximum Likelihood Estimation (MLE) is mostly used to find parameters of the Kernel (or Correlation Function) and this may incorporate the choice of Kernel itself. Whether to call this "chosen by the algorithm itself" depends on what you mean by that. MLE is still an optimization problem, and one that (with regards to Kernel parameters) has to be solved numerically.

You could as well use cross-validation (CV) for this purpose. Which in some cases may work better: Bachoc [1] states that CV works better if the Kernel function is misspecified. In other cases MLE will usually be better, and computationally less expensive.

Also, I have rarely seen the Kernel choice implemented in an automated way. That may be because the resulting optimization problem has a mixture of categorical (choice of Kernel) and real-valued (Kernel parameters) parameters.

[1]:Francois Bachoc. Cross Validation and Maximum Likelihood estimation of hyper-parameters of Gaussian processes with model misspecification. Computational Statistics and Data Analysis, Elsevier, 2013, 66, pp.55-69. Link: https://hal.inria.fr/hal-00905400/document