Proving the predictive distribution in "Pattern Recognition and Machine Learning" by Bishop

Question

I'm studying this book as it is the teaching material for one of the courses I'm taking now. In page 31, it says the predictive distribution is given by the integral of the likelihood function times the posterior probability over weights w:

$p(t \mid x, \mathbf{x}, \mathbf{t})=\int p(t \mid x, \mathbf{w}) p(\mathbf{w} \mid \mathbf{x}, \mathbf{t}) \mathrm{d} \mathbf{w}$

Moreover, the equation above transforms all the way into:

$p(t \mid x, \mathbf{x}, \mathbf{t})=\mathcal{N}\left(t \mid m(x), s^{2}(x)\right)$ with mean and variance given by

$m(x)=\beta \boldsymbol{\phi}(x)^{\mathrm{T}} \mathbf{S} \sum^{N} \boldsymbol{\phi}\left(x_{n}\right) t_{n}$,

$s^{2}(x)=\beta^{-1}+\phi(x)^{\mathrm{T}} \mathbf{S} \phi(x)$,

and the matrix S is $\mathbf{S}^{-1}=\alpha \mathbf{I}+\beta \sum_{n=1}^{N} \phi\left(x_{n}\right) \phi(x)^{\mathrm{T}}$

Alpha and beta is hyper parameters controlling the precision of the distributions, and phi is the basis function.

My question is how can I work out the integral, and why can it be expressed in the form of another Gaussian equation as shown above?

Answer 1

You can workout the integral by noting that $p(t)$ is the marginal distribution between $p({\bf w})$ and $p(t\vert{\bf w})$.

More specifically, if you refer to page 93 of the book, it says

Given a marginal Gaussian distribution for $\bf x$ and a conditional Gaussian distribution for $y$ given $\bf x$ in the form $$p({\bf x}) = \mathcal{N}({\bf x} |\boldsymbol\mu , \boldsymbol\Lambda^{-1})$$ $$p({\bf y}\vert {\bf x}) = \mathcal{N}({\bf y} | {\bf Ax} + b, {\bf L}^{− 1})$$ The marginal distribution of ${\bf y}$ [...] [is] given by $$p({\bf y})=\mathcal{N}({\bf y} | {\bf A} \boldsymbol\mu + b, {\bf L}^{−1} + {\bf A}\boldsymbol\Lambda^{−1}{\bf A}^T)$$

By making the correct substitutions for $\boldsymbol\mu$, $\boldsymbol\Lambda$, $b$, and $\bf A$, you get the predictive distribution.