What is the Gold Standard for Evaluating the Posterior of a Bayesian Regression Model?

Question

Let me explain my meaning & the context:

• I mean evaluating the correctness of the posterior (e.g. for approximate Bayesian inference methods).
• I care mostly about Bayesian deep learning, I'd like a method that scales to very high dimensions & parameter count.
• There are many methods available for classification (e.g. Brier score), I'm asking for something comparable for regression however.

One promising idea is the posterior predictive check but tbh I'm not sure how to implement this in a practical manner given that most Bayesian Models output samples rather than explicit pdfs...

Any ideas?

Answer 1

The method of choice to evaluate probabilistic predictions is a proper scoring rule. The Brier score is one such, for the special case of 0-1 outcomes - that is, classification. However, there are many proper scoring rules that apply to numerical predictions, whether interval or ordinal scaled. The most commonly used ones are probably the log score (which is also applicable to classification, and of course is essentially just the log-likelihood), or the continuous ranked probability score (CRPS).

The scoring-rules tag wiki contains pointers to literature describing proper scoring rules, and perhaps most importantly, to a paper by Merkle & Steyvers (2013) on how to choose between the different possibilities.

Answer 2

There are multiple senses of "correct" you may want to invoke.

A) - good predictions. If by "correct" you mean "has useful predictions", than cross validation (or better: validation on new external data), using a proper scoring rule is probably the way to go. A good proper scoring rule is the log predictive density - this is what is implemented in the loo package which provides ways to approximate leave-one-out cross-validation in an efficient manner. See also A. Vehtari's excellent Cross validation FAQ

Often it also makes sense to evaluate predictions by a scoring rule derived from your application (e.g. in business contexts you may want to try to evaluate the dollar cost/benefit of predictions).

B) - matches data. If by "correct" you mean "matches the true data-generating process", then posterior predictive checks (PPC) are a good way to go. Note that in practice PPCs are almost exclusively used with samples, so there should be no problem. You just take the samples of a quantity from your posterior and compare this distribution to the observed value of the quantity. If the observed value is extreme in respect to the posterior, it signals a problem.

See the section on PPCs in the Bayesian workflow preprint for some simple examples and some context (full disclosure, I am a co-author). The bayesplot package has further examples in their vignette on PPC.

C) correct computation. If by "correct" you mean "the computed posterior matches the theoretical posterior for the given model", then a strong (but computationally expensive) check is provided by simulation-based calibration checking (SBC, see Modrák et al., Bayesian analysis for a recent take on this, disclosure: I am the author). Checking prediction calibration (e.g. that 95% posterior credible interval contains the true value in 95% of the cases) can be understood as a special case of SBC - happy to elaborate more if that's what you are interested in. The linked paper surveys other possibilities. Since it was written, some closely related checks/extensions by Lemos et al. 2023, Yao & Domke 2023 came out.

Note that all of the three senses of "correct" are largely independent and one does not imply any of the others, i.e. your model may satisfy any arbitrary subset of those conditions.

Answer 3

Adding to Stephan Kolassa's answer, there are some (strictly) proper scoring rules for multivariate distributions which are particularly easy to compute (or rather one should say 'estimate') from realizations of the predictive distribution. One of them is the energy score (strictly speaking, this is the member of the energy score family with parameter $\beta = 1$, but this is often ignored) given by $$ S(F, y) = \mathbb{E}_F \Vert X - y \Vert - \frac{1}{2} \mathbb{E}_F \Vert X- X' \Vert $$ where $X$ and $X'$ are independent random vectors with distribution $F$. The energy score is in fact a generalization of the continuous ranked probability score (CRPS) to the multivariate case and it is strictly proper. For details see Strictly Proper Scoring Rules, Prediction, and Estimation by Gneiting and Raftery.

For more practical background you can have a look at Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities by Scheuerer and Hamill, who compare the energy score to other possible choices, in particular their Variogram score. Additionally, there is a method to compute the energy score in the R-package scoringRules, just search the manual for 'energy'.