3
$\begingroup$

I have been trying to figure out how to get the marginal likelihood of a GP model. I am working on a regression problem, where my target is $y$ and my inputs are denoted by $x$. The model is $y_i=f(x_i)+\epsilon$, where $\epsilon \sim N(0,\sigma^2)$

Therefore, the marginal likelihood:

$p(y|x) = \int p(y|f,x)p(f|x)df$, where

$p(y|f,x)\sim N(f,\sigma^2I)$ and

$p(f|x)\sim N(0,K)$

I know that the result should be $N(0,K+\sigma^2I)$. However, I am not sure why this is true. If anyone can recommend where I can find the proof or give me a hint I would really appreciate it.

$\endgroup$
1
  • $\begingroup$ I propose you two solutions : i) use the moment generating functions (the easiest) or ii) integrate by starting with the 1D case $\endgroup$
    – beuhbbb
    Mar 2, 2016 at 10:47

1 Answer 1

2
$\begingroup$

Say you observe $\{(x_i, y_i)\}_{i=1}^n$. Let $X:=(x_1|\cdots |x_n)$, $\mathbf{f} := (f_1,\ldots, f_n)$ and $Y:=(y_1,\ldots, y_n)$. We have

$$\begin{eqnarray} p(y|f, x) &=& p(y|f) = \mathcal{N}(y|f, \sigma^2) \\ p(\mathbf{f}|X) &=& \mathcal{N}(\mathbf{f}| 0, K) \\ p(Y|X) &=& \int p(Y|\mathbf{f}) p(\mathbf{f}|X) d\mathbf{f} = \int p(\mathbf{f}|X) \prod_{i=1}^n p(y_i|f_i) d\mathbf{f} \\ &=& \int \mathcal{N}(\mathbf{f}| 0, K) \mathcal{N}(Y|\mathbf{f}, \sigma^2 I) d\mathbf{f} \\ &=& \mathcal{N}(Y|0, K+\sigma^2I). \end{eqnarray}$$

You get the result because of the following property of the multivariate normal distribution. If $p(a|b)= \mathcal{N}(a|Ab, S)$ and $p(b) = \mathcal{N}(b|\mu, \Sigma)$, then

$$\begin{eqnarray} p(a) = \int p(a|b)p(b) db = \mathcal{N}(a|A\mu, A\Sigma A^\top + S). \end{eqnarray}$$

If you wonder why the last result holds, I think this is another separate question that is independent of a Gaussian process.

$\endgroup$
2
  • $\begingroup$ I do wonder why that last result holds, and I have been unable to find further information. Can you give me a hint of how this result is called, or where I can read in detail about it? $\endgroup$ Jan 30, 2019 at 13:52
  • $\begingroup$ Rough explanation: p(a,b) is a joint Gaussian distribution. It is known that marginal distribution of a joint Gaussian is a Gaussian. For details, one source of reference is section 2.3.2, page 88 of "Pattern Recognition and Machine Learning" book which you can now download for free. microsoft.com/en-us/research/people/cmbishop/#!prml-book $\endgroup$
    – wij
    Jan 31, 2019 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.