I'm new to Bayesian analysis, and still getting acquainted to some of the basic ideas. I'm working through Gaussian Processes for Machine Learning. Stepping through Chapter 2 I'm very confused with this Figure 2.1, page 10 showing an example of a Bayesian linear model. Figure 2.1 from Gaussian Processes for Machine Learning

Here (a) shows the prior, p(w)~N(0,I), (b) shows training points, (c) shows "contours of the likelihood p(y|X,w) ... assuming a noise level of sigma_n = 1" and (d) shows the posterior.

What's confusing me about this plot is that the likelihood function given by R&W is $$ p(\mathbf{y}|X,\mathbf{w}) = \mathcal{N}(X^T\mathbf{w},\sigma_n^2 I) $$ where $\sigma_n=1$. So could anyone help me understand the following?

  1. Since the mean of the likelihood distribution is itself dependent upon the parameters $\mathbf{w}$, what is used for the mean in this plot (c)?
  2. The covariance matrix for the likelihood is stated as $\sigma_n^2 I$, so I would expect the distribution to be symmetric, but in the plot it clearly isn't. Why?

1 Answer 1


When they write
$$ \mathcal{N}(X^T\mathbf{w},\sigma_n^2 I) \propto \exp\left[-\frac{1}{2\sigma^2}(y-X^T\mathbf{w})^T (y-X^T\mathbf{w})\right] $$ they are interpreting it as a function of $\mathbf{w} = (w_1,w_2)^T$, not $y$. That's the key point.

Understanding the plot comes after you rewrite that same function as something proportional to $$ \exp\left[-\frac{1}{2\sigma^2}(\mathbf{w} - [(XX^T)^{-1}Xy])^T(XX^T)(\mathbf{w} - [(XX^T)^{-1}Xy]) \right]. $$ So your mean of that (c) plot, in this case, would be the regular OLS estimator. And notice that the covariance matrix is no longer diagonal: it's equal to $\sigma^2(XX^T)^{-1}$. This is the variance of the OLS estimator.

This is reassuring because a) the formulas are familiar, and b) it makes sense when we remember that the regular OLS estimator maximizes the likelihood (ignoring $\sigma^2$) when it minimizes the least squares criterion.

The last plot, when you take into account the prior, scooches the mode closer to the origin (which is where the prior was centered).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.