Interpreting Rasmussen and Williams likelihood function plot

Question

I'm new to Bayesian analysis, and still getting acquainted to some of the basic ideas. I'm working through Gaussian Processes for Machine Learning. Stepping through Chapter 2 I'm very confused with this Figure 2.1, page 10 showing an example of a Bayesian linear model.

Here (a) shows the prior, p(w)~N(0,I), (b) shows training points, (c) shows "contours of the likelihood p(y|X,w) ... assuming a noise level of sigma_n = 1" and (d) shows the posterior.

What's confusing me about this plot is that the likelihood function given by R&W is $$ p(\mathbf{y}|X,\mathbf{w}) = \mathcal{N}(X^T\mathbf{w},\sigma_n^2 I) $$ where $\sigma_n=1$. So could anyone help me understand the following?

1. Since the mean of the likelihood distribution is itself dependent upon the parameters $\mathbf{w}$, what is used for the mean in this plot (c)?
2. The covariance matrix for the likelihood is stated as $\sigma_n^2 I$, so I would expect the distribution to be symmetric, but in the plot it clearly isn't. Why?

Answer 1

When they write
$$ \mathcal{N}(X^T\mathbf{w},\sigma_n^2 I) \propto \exp\left[-\frac{1}{2\sigma^2}(y-X^T\mathbf{w})^T (y-X^T\mathbf{w})\right] $$ they are interpreting it as a function of $\mathbf{w} = (w_1,w_2)^T$, not $y$. That's the key point.

Understanding the plot comes after you rewrite that same function as something proportional to $$ \exp\left[-\frac{1}{2\sigma^2}(\mathbf{w} - [(XX^T)^{-1}Xy])^T(XX^T)(\mathbf{w} - [(XX^T)^{-1}Xy]) \right]. $$ So your mean of that (c) plot, in this case, would be the regular OLS estimator. And notice that the covariance matrix is no longer diagonal: it's equal to $\sigma^2(XX^T)^{-1}$. This is the variance of the OLS estimator.

This is reassuring because a) the formulas are familiar, and b) it makes sense when we remember that the regular OLS estimator maximizes the likelihood (ignoring $\sigma^2$) when it minimizes the least squares criterion.

The last plot, when you take into account the prior, scooches the mode closer to the origin (which is where the prior was centered).