1
$\begingroup$

I've been studying Gaussian processes (GP), this resource implements GP from scratch and has been very useful in visualizing what's happening. So far all has made sense to me except for the below equation (eq 11 in link), the log marginal likelihood of the GP:

$$ -1/2 [Y^{T} K_y^{-1}Y] -1/2 [log(|K_y|)] - N/2[log(2 \pi)]$$

The author explains that this step is necessary to optimize the hyperparameters of the kernel function. I've used some algebra and found that this is simply the log multivariate gaussian PDF:

$$ \frac{exp(-1/2 Y^{T}K_y^{-1}Y)}{|K_y|^{1/2}(2\pi)^{N/2}} $$

My question is- what has been marginalized out? I gather it's either the old data (training) or the new data (at inference). But I'm not sure which. Intuition and context lead me to believe that we're optimizing kernel hyperparamters for the new data (else why not just accomplish this step during in the prior?)

However, the author defines {$K,K_*, K_{**}$} as $K(old, old)$, $K(old, new)$, and $K(new, new)$. I'm not sure which of the above $K_y$ is supposed to represent, what has been marginalized out, and some intuitions as to why it's been marginalized out. Could anyone comment on the what/why of what's going on here?

$\endgroup$

1 Answer 1

2
$\begingroup$

$K_y$ represents $K(old, old)$. You are marginalizing out the Prior over function values, which is explained in the "Prior" section of the blog. By marginalizing out the prior over function values you are able to optimize parameters of kernel function(in the case of RBF kernel it is $l$ and $\sigma_f$) based on the information you acquire from the training data.

I believe this blog is a review of the some chapters of "Gaussian Processes for Machine Learning by Carl Edward Rasmussen and Christopher K. I. Williams". The book explains the concepts clearly.

$\endgroup$
3
  • $\begingroup$ This optimization technique seems akin to MLE. I'm not seeing anything particularly Bayesian in the author's write up, other than the use of words, prior & posterior. In a Bayesian design, would the parameters, {$l, \sigma_f$} be sampled? (Or am I confused?) $\endgroup$
    – jbuddy_13
    Jan 2, 2021 at 15:46
  • $\begingroup$ Also, the author uses "95% confidence interval" often, which is from the frequentist paradigm. $\endgroup$
    – jbuddy_13
    Jan 2, 2021 at 15:54
  • 1
    $\begingroup$ I think you are confused because author does not derive the results in functions 4 and 5. Those results deriven from posterior over function values, with the assumption poserior follows a gaussian distrb. Parameters $\{l, \sigma_f \}$ are estimated by MLE because they are required to calculate mean and covariance which are obtained from posterior. "95% confidence interval" can be used in bayesian setting too as long as you are able to estimate distrb, difference is in the estimation of distrb. not the usage of confidence interval. Chapter 2 of the book explains your questions in detail. $\endgroup$ Jan 2, 2021 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.