I am reading Gaussian Process for Machine Learning Chapter 2 and I have a question about the subfigure (c) in figrue 2.1.

The slope is much more "well determined" than the intercept.

I was wondering why is it. So I derive the distribution for likelihood.We know that the linear model is

\begin{gather*} f( x) =\mathbf{x^{\top } w} ,\ \quad y=f(\mathbf{x}) +\epsilon \\ \epsilon \sim \mathcal{N}\left( 0,\sigma ^{2}_{n}\right) \end{gather*} where $\mathbf{x} \in \mathbb{R}^{d}$ and $\mathbf{w} \in \mathbb{R}^{d}$. If we get $n$ data points, we can form a matrix $X\in \mathbb{R}^{d\times n} ,\mathbf{y} \in \mathbb{R}^{n}$. And the likelihood for this linear model

\begin{align*} p(\mathbf{y} |X,\mathbf{w} ) & =\prod ^{n}_{i=1} p(y_{i} |\mathbf{x}_{i} ,\mathbf{w} )\\ & \varpropto \exp (-\frac{1}{2\sigma ^{2}_{n}} |\mathbf{y} -X^{\top }\mathbf{w} |^{2} )\\ & \varpropto \exp (-\frac{1}{2\sigma ^{2}_{n}}\left(\mathbf{y} -X^{\top }\mathbf{w}\right)^{\top }\left(\mathbf{y} -X^{\top }\mathbf{w}\right)\\ & \varpropto \exp\left( -\frac{1}{2\sigma ^{2}_{n}}\left(\mathbf{w}^{\top } XX^{\top }\mathbf{w} -2y^{\top }\left( X^{\top }\mathbf{w}\right)\right)\right)\\ & \varpropto \exp\left( -\frac{1}{2\sigma ^{2}_{n}}\left(\mathbf{w}^{\top } XX^{\top }\mathbf{w} -2\mathbf{w}^{\top }( Xy)\right)\right)\\ & \varpropto \exp\left( -\frac{1}{2\sigma ^{2}_{n}}\left(\mathbf{w} -\hat{\mathbf{w}}\right)^{\top } XX^{\top }\left(\mathbf{w} -\mathbf{\hat{w}}\right)\right) \end{align*}

So the likelihood distribution is $p(\mathbf{w}) \sim \mathcal{N}\left(\hat{\mathbf{w}} =\left( XX^{\top }\right)^{-1}( Xy) ,\left( XX^{\top }\right)^{-1}\right)$

where $\hat{\mathbf{w}} =\left( XX^{\top }\right)^{-1}( X\mathbf{y})$ is the least square solution.

So I was thinking about how to interpret the $X^\top X$ term to get some sense of why the figure looks in this shape(the axis length for $\mathbf{w}_1$ is longer than $\mathbf{w}_2$ and its axis aligns with the coordinate axis.

  • $\begingroup$ What is the question? Note that the likelihood as a function of $\mathbf{w}$ is proportional to a Gaussian density on $w$, but that this does not make $\mathbf{w}$ a random variable. It is more relevant to see it as a density on $\hat{\mathbf{w}}$. $\endgroup$ – Xi'an Apr 2 '18 at 8:09
  • $\begingroup$ @Xi'an want to get some intuition about why the contours of the likelihood function in the figure look like this(it is fatter in $\mathbf{w}_1$ and thinner in $\mathbf{w}_2$), that's the reason why I write the likelihood function. After writing that down, I have found out that the center of the contour should be $\hat{\mathbf{w}}$. But I was wondering what the covariance part means. Yes, in likelihood function $\mathbf{w}$ indeed isn't a random variable, I think modified $p(\mathbf{w})$ to $f(\mathbf{w})$ will have less confusion. $\endgroup$ – David Apr 2 '18 at 8:28
  • $\begingroup$ The shape of the Gaussian regression likelihood is elliptic wrt to the quadratic form derived from $\mathbf{X}^\text{T}\mathbf{X}$ so the scales of the regressors induce the variability in the range of the likelihood function indeed. $\endgroup$ – Xi'an Apr 2 '18 at 8:33
  • $\begingroup$ I uploaded the input points for the likelihood function. As the variance of the likelihood is related to $\mathbf{X}^\top \mathbf{X}$, does it mean that if I move all input points upward along y-axis 1 unit, the variance of the likelihood will change? $\endgroup$ – David Apr 2 '18 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.