# What is the Gold Standard for Evaluating the Posterior of a Bayesian Regression Model?

Let me explain my meaning & the context:

• I mean evaluating the correctness of the posterior (e.g. for approximate Bayesian inference methods).
• I care mostly about Bayesian deep learning, I'd like a method that scales to very high dimensions & parameter count.
• There are many methods available for classification (e.g. Brier score), I'm asking for something comparable for regression however.

One promising idea is the posterior predictive check but tbh I'm not sure how to implement this in a practical manner given that most Bayesian Models output samples rather than explicit pdfs...

Any ideas?

– Dave
Commented Jul 13 at 19:57
• It is simply false that Brier score only applies to pure classification. After all, $(1-0.8)^2$, for example, makes perfect sense.
– Dave
Commented Jul 13 at 19:59
• I recommend editing the question to reflect this explicitly, including a modification of the title to clarify what you mean.
– Dave
Commented Jul 13 at 20:53
• I recommend looking at Gelman et al 2013 and Vehtari et al 2024. Commented Jul 13 at 21:17
• From a calibration perspective, I recommend looking at Angelopoulos & Bates 2022 which have a section on combining Bayesian inference with conformal prediction. Commented Jul 13 at 21:42

The method of choice to evaluate probabilistic predictions is a proper scoring rule. The Brier score is one such, for the special case of 0-1 outcomes - that is, classification. However, there are many proper scoring rules that apply to numerical predictions, whether interval or ordinal scaled. The most commonly used ones are probably the log score (which is also applicable to classification, and of course is essentially just the log-likelihood), or the continuous ranked probability score (CRPS).

The tag wiki contains pointers to literature describing proper scoring rules, and perhaps most importantly, to a paper by Merkle & Steyvers (2013) on how to choose between the different possibilities.

• Thanks! But I'm concerned that you mention it's the log-likelihood. Does this method work if you only have samples of the posterior rather than an explicit function (e.g. MCMC)? I'm concerned because high dimensional pdf estimation is tricky (e.g. KDE doesn't scale well). Commented Jul 13 at 22:11
• You have a point. In principle, proper scoring rules will by definition be minimized in expectation by the correct distribution... which will likely translate into some asymptotic statement if all we have is samples from our predictive distribution. No, I don't know of any work in this direction, sorry! Commented Jul 14 at 12:39
• It's ok. I just learned in my case of mean-field variational inference for deep learning, you can apparently prove the distribution of posterior predictions (model outputs) will be gaussian which solves the problem for me. Commented Jul 14 at 16:48
• That is not so! That is an asymptotic result, not a finite sample result! Commented Jul 14 at 17:24
• Good point it's just a CLT approximation... But I'm not sure what else to do? Commented Jul 14 at 18:48

There are multiple senses of "correct" you may want to invoke.

A) - good predictions. If by "correct" you mean "has useful predictions", than cross validation (or better validation on new external data), using a proper scoring rule is probably the way to go. A good proper scoring rule is the log predictive density - this is what is implemented in the loo package which provides ways to approximate leave-one-out cross-validation in an efficient manner. See also A. Vehtari's excellent Cross validation FAQ

Often it also makes sense to evaluate predictions by a scoring rule derived from your application (e.g. in business contexts you may want to try to evaluate the dollar cost/benefit of predictions).

B) - matches data. If by "correct" you mean "matches the true data-generating process", then posterior predictive checks (PPC) are a good way to go. Note that in practice PPCs are almost exclusively used with samples, so there should be no problem. You just take the samples of a quantity from your posterior and compare this distribution to the observed value of the quantity. If the observed value is extreme in respect to the posterior, it signals a problem.

See the section on PPCs in the Bayesian workflow preprint for some simple examples and some context (full disclosure, I am a co-author). The bayesplot package has further examples in their vignette on PPC.

C) correct computation. If by "correct" you mean "the computed posterior matches the theoretical posterior for the given model", then a strong (but computationally expensive) check is provided by simulation-based calibration checking (SBC, see Modrák et al., Bayesian analysis for a recent take on this, disclosure: I am the author). Checking prediction calibration (e.g. that 95% posterior credible interval contains the true value in 95% of the cases) can be understood as a special case of SBC - happy to elaborate more if that's what you are interested in. The linked paper surveys other possibilities. Since it was written, some closely related checks/extensions by Lemos et al. 2023, Yao & Domke 2023 came out.

Note that all of the three senses of "correct" are largely independent and one does not imply any of the others, i.e. your model may satisfy any arbitrary subset of those conditions.

• Thanks for the variety of options. Regarding SBC is the purpose to validate all credible intervals simultaneously? I had an idea about how to approximate this but it's confusing in high dimensions what the best approach is. Commented Jul 14 at 21:28
• The easiest (and typical) way is to look at various univariate projections/functions (test quantities) one at a time. The approach by Yao is a clever way to look at the full joint, but may require much more computation. The Lemos way is to choose a distance metric(s) which once again make the problem 1D. I think, but so far cannot prove that the "distance metrics" in Lemos approach actually correspond 1-to-1 to "test quantities" in SBC. Commented Jul 15 at 5:03
• And yes, SBC validates all credible intervals simultaneously - the idea is that you look at the rank of the true (simulated) value within the posterior. The distribution of ranks is uniform if and only if all credible intervals are calibrated. Commented Jul 15 at 5:22
• Omg that's so cool! I literally came up with that idea on my own. It's awesome to know that other people are using it too. I tried using the Lemos approach, but ya I'm not sure ofc that it is 100% valid. Commented Jul 15 at 19:03
• The problem I had is that probably the "most correct way" (maybe Yao's approach?), would be to estimate the density of the predicted distribution then sort them by density instead of distance. But the only way I could think to do this is with KDE which apparently doesn't scale very well to high dimensions... Commented Jul 15 at 19:06