10
$\begingroup$

I am reviewing textbooks for our new undergraduate course in Bayesian Statistical Methods. In chapter 7 of Ben Lambert's book, A Student's Guide to Bayesian Statistics, he states

Because of the two sources of uncertainty included in our model – the parameter uncertainty and sampling variability – the uncertainty of the Bayesian predictive distribution is typically greater than the Frequentist equivalent.  This is because the Frequentist approach to forecasting typically makes predictions based on a point estimate of a parameter (typically the maximum likelihood value).  By ignoring any uncertainty in the parameter’s value, the Frequentist approach produces predictive intervals that are overly confident.

[emphasis mine]

Then in the chapter summary, it is restated:

By including our epistemic uncertainty in parameter values as part of a forecast, this Bayesian approach provides a better quantification of uncertainty than the equivalent Frequentist methods.

Without going into more detail about how this book is written to suggest that everything Bayesian is good and correct (and thus all things Frequentist are bad and incorrect), I found myself needing to push back on this claim.

While I believe I know the answer to the titular question, perhaps my understanding is incorrect...and I wanted to put it here for feedback from this community.

As an example of how I believe this assertion in the book to be incorrect, I will use an example from multiple regression. If you wish to estimate a value from a model, you have the following $$\hat{y}_o = \beta_0 + \beta x_o$$ (where $\beta$ and $x_o$ may be vectors). If you wish to use this estimate as the conditional mean given the values of $X=x_o$, then $\hat{y}_o$ would be the point estimate, and the confidence interval for the conditional mean would be $$\hat{y}_o + t_\text{c.v.} \cdot \hat{\sigma} \sqrt{x_o^T (X·X^T)^{-1} x_o}$$ (for the appropriate critical value for the context and desired confidence level).

However, if you want the prediction (confidence) interval, you would use the following: $$\hat{y}_o + t_\text{c.v.} \cdot \hat{\sigma} \sqrt{1 + x_o^T (X·X^T)^{-1} x_o}$$ ...and unless I am wrong, the radicand captures the "epistemic uncertainty" that that author claims is not present. The 1 models the variability from the distribution and the second term models the variability from the parameter estimate (as it is not assumed to be a fixed value for prediction interval estimation).

Again, if I am incorrect or there are other contexts where this critique of the Frequentist approach is indeed valid...please share.

$\endgroup$
14
  • 3
    $\begingroup$ Would love to see an example that showed in what sense the text author believes frequentist prediction intervals are overly confident. $\endgroup$ Jun 30, 2023 at 7:20
  • 4
    $\begingroup$ Many of the "equivalent frequentist methods" for standard statistical procedures have been shown to be mathematically the same as adopting certain Bayes priors. (This is a standard way to prove admissibility.) Usually those priors are quite diffuse, sometimes referred to as "uninformative." The quotations, then, are not just misleading: they are even mathematically incorrect. $\endgroup$
    – whuber
    Jun 30, 2023 at 16:08
  • 3
    $\begingroup$ By the way, although I'm not against Bayesian methodology or philosophy, the question shows one of many examples for annoying "Bayesian propaganda" that doesn't just properly explain the workings and also potential pitfalls of Bayesian analysis but spends much space and energy bashing and ridiculing some caricature of frequentist statistics in order to state that the Bayesian approach is superior and frequentists are idiots. My dear Bayesians, this kind of communication may well backfire and put off people who'd otherwise have an open mind for what you have on offer! $\endgroup$ Jul 1, 2023 at 11:04
  • 3
    $\begingroup$ I'm not sure what's so "desperate" in Lambert's description. Regardless, I think it can't be emphasized enough to refrain from the pseudo-Bayesian interpretation of confidence intervals as "capturing the epistemic uncertainty" about $\beta$. If that's what you want your interval estimate to mean, you have to start with a prior and use Bayes' rule at some point. $\endgroup$
    – Durden
    Jul 1, 2023 at 19:43
  • 2
    $\begingroup$ One may call the notion (expressed in the original question) of confidence intervals "capturing the epistemic uncertainty" about $\beta$ as the misinterpretation of CIs as Bayesian credible intervals. The many older posts linked above explain at length why this notion is misguided. $\endgroup$
    – Durden
    Jul 1, 2023 at 22:44

2 Answers 2

18
$\begingroup$

Does the Frequentist approach to forecasting ignore uncertainty in the parameter's value?

No! Or rather, it shouldn't (and normally doesn't), but of course an individual or some particular methodology might leave it out in some situation or other, either deliberately - e.g. because it's known to be so small as to not matter because sample size was truly huge; where bias is a much bigger concern than parameter uncertainty - or in ignorance (either of the need for including it, or of how to do so).

[Off the top of my head the only time I've noticed it happen was when someone omitted parameter uncertainty with a forecast from a GLM, out of one of those forms of ignorance.]

It's always important to be clear about what you're predicting.

A CI for a conditional mean would consider uncertainty in the parameter value, while a PI would consider both that parameter uncertainty and observation noise. I have a couple of times seen people forget the observation noise (i.e. confusing a CI for a PI) rather than parameter uncertainty.

You use a regression example, for which the prediction interval (clearly) has two terms in the variance, and it's easy to see that they account for both the observation noise and parameter uncertainty.

I haven't read Lambert's book, but from the quote he appears to misrepresent the normal situation in frequentist forecasting.

$\endgroup$
4
  • 1
    $\begingroup$ The obligatory reminder that we have to be careful not to interpret confidence intervals as representing "uncertainty in the parameter value". $\endgroup$
    – Durden
    Jun 30, 2023 at 13:27
  • 2
    $\begingroup$ @Durden On the contrary, measuring uncertainty in parameter estimates is one of the main reasons why we construct confidence intervals. $\endgroup$ Jun 30, 2023 at 13:58
  • 4
    $\begingroup$ Uncertainty in the estimates, not uncertainty in the parameter itself. Those are not necessarily the same. $\endgroup$
    – Durden
    Jun 30, 2023 at 14:16
  • 2
    $\begingroup$ If you're talking about the parameter itself changing, that's now just talking about a different kind of error, specifically model specification error, which you can get under either paradigm. If you're not talking about the parameter changing (i.e. if it is just fixed-but-unknown) then the CI does take account of the uncertainty in its value. $\endgroup$
    – Glen_b
    Jun 30, 2023 at 19:27
3
$\begingroup$

I would say that the Bayesian approach forces the practitioner to take the uncertainty in the parameters into account. The posterior predictive distribution automatically embeds both observation noise and parameter uncertainty. For conjugate distributions the posterior predictive can be found in closed form. In the other cases it is well defined theoretically but hard to find/compute. A large number of approaches (MCMC, HMC, etc), tools and probabilistic programming languages (STAN, pymc, pyro, etc) have been developed specifically to address this sort of questions.

In the frequentist approach the are ways to account for the uncertainty in the parameters, as in the linear regression example in the OP. However there are two issues. One is that they are only available in closed form for a limited number of cases (like linear regression) but not for most cases (even simple deviations from LR like with Huber loss or lasso regularisation). The second issue (probably related to the first one) is that in practice you'll find that in many cases (I would dare to say in most cases) practitioners actually make the mistake of fitting some parameters to some data (typically using maximum likelihood) and then use the fitted model (with point estimate of the parameters) to make predictions on new data. There are methods to make predictions with an associated interval (e.g. with "conformal predictions" techniques) but they are often add-ons, rather than part of the model like in the Bayesian approach. On the other hand, since they are less reliant on assumptions than Bayesian techniques, they may be more robust and accurate in practice.

In other words my impression is that it is virtually impossible to "forget" to account for parameters uncertainty in Bayesian statistics while it's possible (and often done "in the field") in frequentist statistics.

$\endgroup$
2
  • 3
    $\begingroup$ Practitioners may also "estimate the prior from the data" when doing Bayesian analyses and in this way underestimate the Bayesian uncertainty. $\endgroup$ Jul 1, 2023 at 11:00
  • $\begingroup$ @ChristianHennig Good point! Btw, I'm not implying the Bayesian approach is error-proof. I was just focusing on the specific issue of whether the parameters uncertainty is accounted for (in a natural way). Actually, if I had to build a device making important decisions and badly needed prediction accuracy estimates, I would not blindly trust a Bayesian approach (heavily reliant on making the correct assumptions) and would probably go for the frequentist method called "conformal predictions" (which is almost assumption-free) $\endgroup$
    – Luca Citi
    Jul 1, 2023 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.