I'm studying a book about Gaussian processes in machine learning, and I do not know how to exactly compute the predictive distribution. In this question: Understanding the predictive distribution in gaussian linear regression said it is like $(1)$, but I found in Bishop's Pattern Recognition and ML that this predictive distribution is like $(2)$:
\begin{align} f_*\mid x_*,\, X,\, y\quad &\to\quad \mathcal{N}(\sigma_n^{-2}x^T_* A^{-1}Xy,\; \hspace{13mm} x_*^T A^{-1}x_*) \tag{1} \\ f_*\mid x_*,\, X,\, y\quad &\to\quad \mathcal{N}(\sigma_n^{-2}x^T_* A^{-1}Xy,\; \sigma_n^2 I + x_*^T A^{-1}x_*) \tag{2} \end{align} (That is, with a different variance.) What happened? Why are they not equal?

  • $\begingroup$ looks like minor difference, just a notation. $\endgroup$ – Curious Aug 9 '19 at 20:42
  • $\begingroup$ @Curious there's actually a meaningful difference; see below $\endgroup$ – user20160 Aug 10 '19 at 16:40
  • $\begingroup$ Thanks! Skullgreymon, please include full reference to the Bishop's paper. Anyway, this is interesting question and answer! $\endgroup$ – Curious Aug 10 '19 at 17:43

Short answer

Equations (1) and (2) are different because they give the posterior predictive distribution for different quantities. (1) is for the noiseless linear function output, whereas (2) is for the noisy observed output.

Long answer

Recall that the model is:

$$y_i = w^T x_i + \epsilon_i \quad \epsilon_i \sim \mathcal{N}(0, \sigma^2_n)$$

Each observed output $y_i$ is given by a linear function of the input $x_i$ plus i.i.d. Gaussian noise. The equations listed in the question assume a Gaussian prior on the coefficients $w$, and treat the noise variance $\sigma^2_n$ as a fixed parameter.

Notice that we could also write:

$$y_i = f_i + \epsilon_i$$

where $f_i = w^T x_i$ is the noiseless output of the linear function. This is a latent variable, since it's not directly observed. Rather, we observe the noisy output $y_i$.

Now, suppose we've fit the model to training data $(X,y)$, and want to predict the output for a new set of inputs $x_*$. The posterior of the noiseless function outputs $f_*$ is the Gaussian distribution in equation (1). The derivation is described in chapter 2 of Gaussian processes for machine learning (Rasmusen & Williams 2006), and summarized here.

The posterior of the noisy observed outputs $y_*$ is the Gaussian distribution in equation (2) (but there may be a typo; the variable should be called $y_*$, not $f_*$). Notice that (2) is identical to (1), with the exception that $\sigma^2_n I$ has been added to the covariance matrix. This follows from the fact that the noisy observed outputs are produced by adding independent Gaussian noise to the noiseless function outputs (with mean zero and variance $\sigma^2_n$).

  • $\begingroup$ Exactly. Thank you very much. I had a missunderstanding between the latent function and the noisy observed variable. And I thought that it was the same. $\endgroup$ – Skullgreymon Aug 10 '19 at 16:41
  • $\begingroup$ @Skullgreymon I get the impression that more sources refer to (2) as "the posterior predictive distribution" when describing Bayesian linear regression. So the confusion is quite understandable. GPML might be somewhat unique in emphasizing a latent function in this context. I suppose this makes sense given its focus on Gaussian processes rather than linear regression. $\endgroup$ – user20160 Aug 10 '19 at 16:50

It is the residual noise(error) term, you can refer to Distribution Theory: Normal Regression Models part of https://ocw.mit.edu/courses/mathematics/18-655-mathematical-statistics-spring-2016/lecture-notes/MIT18_655S16_LecNote19.pdf

You may also want to check out the concepts of heteroscedasticity and assumptions of the linear regression

  • 1
    $\begingroup$ I don not understand your point. I think is not about heteroscedasticity, both approaches supposes homocedasticity. And I think it does not make sense that the error term do not appear on the variance of the predictive distribution. $\endgroup$ – Skullgreymon Oct 23 '18 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.