I was reading this article on characterizing the posterior for a simple Bayesian linear regression (Gaussian posterior) when I saw the posterior mean and variance described thusly:

\begin{split} \mu_n &= (X^TX+\sigma^2\Lambda)^{-1}X^Ty,\\ \Sigma_n &= \sigma^2(X^TX+\sigma^2\Lambda)^{-1}. \end{split}

where $\Lambda$ is the precision matrix.

Here is my question: does the term $\sigma^2 \Lambda$ only make the slope smaller? What exactly is going on here?


As the article describes, the posterior you wrote is the result of putting a zero-mean Gaussian prior on the weights. The consequence of this is, indeed, that the weights will be smaller than they would be for maximum likelihood/least squares estimation.

One way to understand this is by considering the shape of the Gaussian prior. It assigns greater probability to weight vectors closer to the mean (i.e. zero). Compare this to the maximum likelihood/least squares case, where the equivalent prior is the improper uniform prior that gives no preference to any weight vector over any other.

Another way to understand this is by thinking about ridge regression. Here, the goal is to find weights that reduce the squared error, but a penalty is placed on the magnitude of the weight vector. The effect of this penalty is to shrink the weights toward zero. The loss function is:

$$\sum_{i=1}^n (y_i - x_i \beta)^2 + \lambda \|\beta\|_2^2$$

where $\beta$ is the weight vector, $x_i$ is the $i$th data point, $y_i$ is the $i$th target value, and $\lambda$ controls the strength of the penalty. As $\lambda$ increases, the weights are increasingly shrunk toward 0. It turns out that ridge regression is equivalent to MAP estimation using a zero-mean Gaussian prior on the weights, whose covariance matrix is a scaled version of the identity matrix (this case is mentioned in the article you linked). The scale of the prior is controlled by $\lambda$--larger penalties correspond to priors that are more strongly concentrated around 0.

  • $\begingroup$ what do you mean by "weights" - as in the two weights in the posterior (the data and the prior)? How could it be that choosing a nonzero prior only decreases the estimated slope? $\endgroup$ – invictus Jan 25 '17 at 18:45
  • $\begingroup$ By weights I mean coefficients of the regression model, i.e. $\beta_1, ...\beta_k$ in the pdf you posted. This is the same as what you're calling slopes, right? $\endgroup$ – user20160 Jan 25 '17 at 19:29
  • $\begingroup$ Ah, yes. So if I understand this correctly, the point of adding the prior $\beta \sim N(0, \Lambda^{-1})$ is to prevent MLE from overfitting? And why is it $\Lambda^{-1}$? I thought the precision was already equal to the reciprocal of the variance, $1/\sigma^2$ $\endgroup$ – invictus Jan 25 '17 at 19:38
  • $\begingroup$ Yes, that's a common reason to do it. Of course, you'll only benefit if the prior is actually suitable for the structure of the data. Otherwise performance can decrease. $\Lambda$ is the inverse covariance matrix, so $\Lambda^{-1}$ is the covariance matrix. $\endgroup$ – user20160 Jan 25 '17 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.