3
$\begingroup$

In Gaussian Process for Machine Learning (Rasmussen and Williams), pg 11, we are given the following predictive distribution:

$$p\left(f_{*} | \mathbf{x}_{*}, X, \mathbf{y}\right)=\int p\left(f_{*} | \mathbf{x}_{*}, \mathbf{w}\right) p(\mathbf{w} | X, \mathbf{y}) d \mathbf{w}$$

We are also given the definition: $f_{*} \triangleq f\left(\mathbf{x}_{*}\right)$ where $f(\mathrm{x})=\mathrm{x}^{\top} \mathrm{w}$ and $y=f(\mathbf{x})+\varepsilon$.

I understand that one may compute the distribution shown above by doing the following:

$$E[f_{*}] = \mathrm{x}^{\top}E[\mathrm{w}]$$ $$V[f_{*}] = \mathrm{x}^{\top}V[\mathrm{w}]\mathrm{x}$$

We then plug in the expressions for the mean and variance of our posterior over w. Essentially, we are computing the mean and variance of a scaled Gaussian random variable as we already established that the posterior distribution over w is Gaussian.

However, I have a problem with the integral shown above. I understand how to arrive at this expression (sum rule, product rule, and conditional independence). However, the $p\left(f_{*} | \mathbf{x}_{*}, \mathbf{w}\right)$ term is bothering me.

I have seen derivations where this term is replaced with $p\left(y_{*} | \mathbf{x}_{*}, \mathbf{w}\right)$. This makes more sense to me as, due to the Gaussian noise, this term would be a Gaussian distribution and the integral of the product of two Gaussian distributions is a Gaussian distribution (provided proper normalisation). However, wouldn't $p\left(f_{*} | \mathbf{x}_{*}, \mathbf{w}\right)$ be a delta function at $f_{*} = \mathrm{x}^{\top}_{*} \mathrm{w}$? Am I missing something here?

Any advice would be greatly welcomed! Cheers!

$\endgroup$
2
  • $\begingroup$ I think you should add the self study tag. $\endgroup$ May 26, 2019 at 3:45
  • $\begingroup$ @MichaelChernick Just added it, thanks for the suggestion. $\endgroup$
    – JKB
    May 26, 2019 at 15:08

1 Answer 1

0
$\begingroup$

It would not be a delta function as we have a probabilistic model over the weight vector w. As they write in (2.4), they put a zero mean Gaussian prior on the vector w. When we incorporate the data, we have to update the prior, and thus obtain a posterior distribution over w. As they show in (2.8), this posterior distribution over w is again a Gaussian, where the parameters are updated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.