Doubts on the derivation of Gaussian Process Regression equations in a paper

Question

I'm reading this paper preprint, and I'm having difficulties following their derivation of the equations for Gaussian Process Regression. They use the setting & notation of Rasmussen & Williams. Thus, additive, zero-mean, stationary and normally distributed noise with variance $\sigma^2_{noise}$ is assumed:

$$y=f(\mathbf{x})+\epsilon, \quad \epsilon\sim N(0,\sigma^2_{noise})$$

A GP prior with zero mean is assumed for $f(\mathbf{x})$, which means that $\forall \ d\in N$, $\mathbf{f}=\{f(\mathbf{x_1}),\dots,f(\mathbf{x_d})\}$ is a Gaussian vector with mean 0 and covariance matrix

$$\Sigma_d=\pmatrix{k(\mathbf{x_1},\mathbf{x_1})& & k(\mathbf{x_1},\mathbf{x_d}) \\ & \ddots & \\k(\mathbf{x_d},\mathbf{x_1})& & k(\mathbf{x_d},\mathbf{x_d}) }$$

From now on, we assume that hyperparameters are known. Then Eq.(4) of the paper is obvious:

$$p(\mathbf{f},\mathbf{f^*})=N\left(0,\pmatrix { K_{\mathbf{f},\mathbf{f}} & K_{\mathbf{f^*},\mathbf{f}} \\K_{\mathbf{f^*},\mathbf{f}} & K_{\mathbf{f^*},\mathbf{f^*}}} \right)$$

Here come the doubts:

1. Equation (5):

   $$p(\mathbf{y}|\mathbf{f})=N\left(\mathbf{f},\sigma^2_{noise}I \right)$$

   $E[\mathbf{f}]=0$, but I guess $E[\mathbf{y}|\mathbf{f}]=\mathbf{f}\neq0$ because when I condition on $\mathbf{f}$, then $\mathbf{y}=\mathbf{c}+\boldsymbol{\epsilon}$ where $\mathbf{c}$ is a constant vector and only $\boldsymbol{\epsilon}$ is random. Correct?

2. Anyway, it's Eq.(6) which is more obscure to me:

   $$p(\mathbf{f},\mathbf{f^*}|\mathbf{y})=\frac{p(\mathbf{f},\mathbf{f^*})p(\mathbf{y}|\mathbf{f})}{p(\mathbf{y})}$$

   That is not the usual form of the Bayes' theorem. Bayes' theorem would be

   $$p(\mathbf{f},\mathbf{f^*}|\mathbf{y})=\frac{p(\mathbf{f},\mathbf{f^*})p(\mathbf{y}|\mathbf{f},\mathbf{f^*})}{p(\mathbf{y})}$$

   I sort of understand why the two equations are the same: intuitively, the response vector $\mathbf{y}$ depends only on the corresponding latent vector $\mathbf{f}$, thus conditioning on $\mathbf{f}$ or on $(\mathbf{f},\mathbf{f^*})$ should lead to the same distribution. However, this is an intuition, not a proof! Can you help me show why

   $$p(\mathbf{y}|\mathbf{f},\mathbf{f^*})=p(\mathbf{y}|\mathbf{f})$$

Answer 1

1. If we fix $\mathbf{f}$, then all uncertainty in $\mathbf{y}$ comes from noise. So for equation (5) in the article we have that given $\mathbf{f}$ we have at each point independent noise with variance $\sigma_{noise}^2$ and mean zero $0$. We add initial mean and get the answer.
2. One way to prove the suggested equality $$ p(\mathbf{y} | \mathbf{f}, \mathbf{f}^*) = p(\mathbf{y} | \mathbf{f}) $$ is to find the distribution at the left hand side and at the right hand side of the quality. Both of them are Gaussian, for the left side we already know the answer. For the right hand side we proceed in a similar way. Let us find the conditional distribution for $(\mathbf{y}, \mathbf{y}^*)$. From the result from the first part we know: $$ p(\mathbf{y}, \mathbf{y}^* | \mathbf{f}, \mathbf{f}^*) = \mathcal{N}((\mathbf{f}, \mathbf{f}^*), \sigma^2_{noise} I). $$ Using probability rules it is easy to integrate out $\mathbf{y}^*$ from $(\mathbf{y}, \mathbf{y}^*)$, as the covariance matrix is diagonal, and vectors $\mathbf{y}$ and $\mathbf{y}^*$ are independent. By doing this we get: $$ p(\mathbf{y} | \mathbf{f}, \mathbf{f}^*) = \mathcal{N}(\mathbf{f}, \sigma^2_{noise} I) = p(\mathbf{y} | \mathbf{f}). $$