Marginal likelihood of a Gaussian Process?

Question

I've been studying Gaussian processes (GP), this resource implements GP from scratch and has been very useful in visualizing what's happening. So far all has made sense to me except for the below equation (eq 11 in link), the log marginal likelihood of the GP:

$$ -1/2 [Y^{T} K_y^{-1}Y] -1/2 [log(|K_y|)] - N/2[log(2 \pi)]$$

The author explains that this step is necessary to optimize the hyperparameters of the kernel function. I've used some algebra and found that this is simply the log multivariate gaussian PDF:

$$ \frac{exp(-1/2 Y^{T}K_y^{-1}Y)}{|K_y|^{1/2}(2\pi)^{N/2}} $$

My question is- what has been marginalized out? I gather it's either the old data (training) or the new data (at inference). But I'm not sure which. Intuition and context lead me to believe that we're optimizing kernel hyperparamters for the new data (else why not just accomplish this step during in the prior?)

However, the author defines {$K,K_*, K_{**}$} as $K(old, old)$, $K(old, new)$, and $K(new, new)$. I'm not sure which of the above $K_y$ is supposed to represent, what has been marginalized out, and some intuitions as to why it's been marginalized out. Could anyone comment on the what/why of what's going on here?

Answer 1

$K_y$ represents $K(old, old)$. You are marginalizing out the Prior over function values, which is explained in the "Prior" section of the blog. By marginalizing out the prior over function values you are able to optimize parameters of kernel function(in the case of RBF kernel it is $l$ and $\sigma_f$) based on the information you acquire from the training data.

I believe this blog is a review of the some chapters of "Gaussian Processes for Machine Learning by Carl Edward Rasmussen and Christopher K. I. Williams". The book explains the concepts clearly.