I'm interested in how we evaluate performance of Bayesian regression (linear, multiple, logistic, etc.) The posterior distribution will capture the relative likeliness of any parameter combination. So a 2D heatmap, for example of B1 and B2 (coefficients) might give us some insight into their relationship.

Recently, a colleague of mine mentioned that the posterior's covariance matrix is effectively "all you need." I want to ask, is this oversimplifying the matter (and even if so) what does the posterior covariance matrix tell you?

My guesses are:

(1) Along the diagonal you get single parameter's variance. The lower the number, the more confidence we have in the estimate. Whereas high variance might indicate that we're less confident in our estimate.

(2) Covariance between parameters might be trickier to interpret. The direction (+/-) of the covariance might give an indication of the nature of the relationship (is an increase in one parameter associated with an increase, decrease or neither in the other.)

(3) The magnitude of the covariance gives me pause. Does a small value imply high confidence in the relationship or little to no association? (Very different meanings!)

(4) I can imagine a situation where the variance of B1 is quite small, so perhaps we're confident in the estimate, whereas the variance of B2 might be rather large, so less confident. I'm not sure how this would affect our understanding of covariance direction and magnitude.

*All the above assumes proper analysis, no multicollinearity, collider bias, etc.

Any thoughts?


1 Answer 1


Under a reasonably vague prior, the Bayesian posterior covariance matrix and the frequentist covariance matrix of the estimates are very similar. Their applications to inference on the regression coefficients are quite similar too. I will refer to this answer here, as it applies equally to the Bayesian posterior covariance matrix: https://stats.stackexchange.com/a/152563/102879. Ignore the last paragraph because it refers to variables, not parameter estimates.

A useful Bayesian addition is that you can think of the covariance matrix in terms of a data scatter of plausible parameter values obtained from posterior simulation (eg, MCMC). Given both the covariance matrix and the posterior means of of the parameters, you can easily hand-draw (in two dimensions) what that scatterplot would look like: Take the posterior means $\pm 3\times$(the square root of the posterior variances) to get the essential ranges of the simulated parameters, and convert the covariance to a correlation to get the tightness and orientation of the scatter. This scatterplot will tell you whether a parameter could plausibly be zero or some other value, depending on degree to which the scatter overlaps the given axis. However, that application involves variances only, and does not involve the covariance.

The covariance enters when you consider joint combinations of the two parameters. For example, in some applications the regression line $E(Y|X=x) = 0 + 1\times x$ may be of interest.

The posterior scatterplot might look like as follows; the hypothetical point $(0,1)$ is indicated with a red "$+$":posterior

From the graph, it appears that the combination $(\beta_0, \beta_1) = (0,1)$ is unlikely, as it appears to be clearly outside of the scatter of plausible values. On the other hand, when considered individually, both $\beta_0=0$ and $\beta_1 = 1$ appear plausible, since the range of plausible values of each parameter individually cover 0 and 1, respectively. Again, the value of the posterior covariance is that it provides information about which combinations of parameter values are plausible.

But it is better to actually simulate from the posterior distribution rather than to try and recreate it from the covariance matrix: Depending on the application, the scatterplot may look quite different than a multivariate normal distribution, and any inferences based on that assumption will be wrong to a degree. Therefore, I disagree with your colleague. The posterior covariance matrix is not all you need. Instead, you need the entire posterior distribution.

  • $\begingroup$ Super helpful, thanks! Might comment back with another thought or two in the near future $\endgroup$
    – jbuddy_13
    Commented Sep 30, 2020 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.