Marginal likelihood of a Gaussian Process

Question

I have been trying to figure out how to get the marginal likelihood of a GP model. I am working on a regression problem, where my target is $y$ and my inputs are denoted by $x$. The model is $y_i=f(x_i)+\epsilon$, where $\epsilon \sim N(0,\sigma^2)$

Therefore, the marginal likelihood:

$p(y|x) = \int p(y|f,x)p(f|x)df$, where

$p(y|f,x)\sim N(f,\sigma^2I)$ and

$p(f|x)\sim N(0,K)$

I know that the result should be $N(0,K+\sigma^2I)$. However, I am not sure why this is true. If anyone can recommend where I can find the proof or give me a hint I would really appreciate it.

Answer 1

Say you observe $\{(x_i, y_i)\}_{i=1}^n$. Let $X:=(x_1|\cdots |x_n)$, $\mathbf{f} := (f_1,\ldots, f_n)$ and $Y:=(y_1,\ldots, y_n)$. We have

$$\begin{eqnarray} p(y|f, x) &=& p(y|f) = \mathcal{N}(y|f, \sigma^2) \\ p(\mathbf{f}|X) &=& \mathcal{N}(\mathbf{f}| 0, K) \\ p(Y|X) &=& \int p(Y|\mathbf{f}) p(\mathbf{f}|X) d\mathbf{f} = \int p(\mathbf{f}|X) \prod_{i=1}^n p(y_i|f_i) d\mathbf{f} \\ &=& \int \mathcal{N}(\mathbf{f}| 0, K) \mathcal{N}(Y|\mathbf{f}, \sigma^2 I) d\mathbf{f} \\ &=& \mathcal{N}(Y|0, K+\sigma^2I). \end{eqnarray}$$

You get the result because of the following property of the multivariate normal distribution. If $p(a|b)= \mathcal{N}(a|Ab, S)$ and $p(b) = \mathcal{N}(b|\mu, \Sigma)$, then

$$\begin{eqnarray} p(a) = \int p(a|b)p(b) db = \mathcal{N}(a|A\mu, A\Sigma A^\top + S). \end{eqnarray}$$

If you wonder why the last result holds, I think this is another separate question that is independent of a Gaussian process.