Does the Frequentist approach to forecasting ignore uncertainty in the parameter's value?

Question

I am reviewing textbooks for our new undergraduate course in Bayesian Statistical Methods. In chapter 7 of Ben Lambert's book, A Student's Guide to Bayesian Statistics, he states

Because of the two sources of uncertainty included in our model – the parameter uncertainty and sampling variability – the uncertainty of the Bayesian predictive distribution is typically greater than the Frequentist equivalent.  This is because the Frequentist approach to forecasting typically makes predictions based on a point estimate of a parameter (typically the maximum likelihood value).  By ignoring any uncertainty in the parameter’s value, the Frequentist approach produces predictive intervals that are overly confident.

[emphasis mine]

Then in the chapter summary, it is restated:

By including our epistemic uncertainty in parameter values as part of a forecast, this Bayesian approach provides a better quantification of uncertainty than the equivalent Frequentist methods.

Without going into more detail about how this book is written to suggest that everything Bayesian is good and correct (and thus all things Frequentist are bad and incorrect), I found myself needing to push back on this claim.

While I believe I know the answer to the titular question, perhaps my understanding is incorrect...and I wanted to put it here for feedback from this community.

As an example of how I believe this assertion in the book to be incorrect, I will use an example from multiple regression. If you wish to estimate a value from a model, you have the following $$\hat{y}_o = \beta_0 + \beta x_o$$ (where $\beta$ and $x_o$ may be vectors). If you wish to use this estimate as the conditional mean given the values of $X=x_o$, then $\hat{y}_o$ would be the point estimate, and the confidence interval for the conditional mean would be $$\hat{y}_o + t_\text{c.v.} \cdot \hat{\sigma} \sqrt{x_o^T (X·X^T)^{-1} x_o}$$ (for the appropriate critical value for the context and desired confidence level).

However, if you want the prediction (confidence) interval, you would use the following: $$\hat{y}_o + t_\text{c.v.} \cdot \hat{\sigma} \sqrt{1 + x_o^T (X·X^T)^{-1} x_o}$$ ...and unless I am wrong, the radicand captures the "epistemic uncertainty" that that author claims is not present. The 1 models the variability from the distribution and the second term models the variability from the parameter estimate (as it is not assumed to be a fixed value for prediction interval estimation).

Again, if I am incorrect or there are other contexts where this critique of the Frequentist approach is indeed valid...please share.

Answer 1

Does the Frequentist approach to forecasting ignore uncertainty in the parameter's value?

No! Or rather, it shouldn't (and normally doesn't), but of course an individual or some particular methodology might leave it out in some situation or other, either deliberately - e.g. because it's known to be so small as to not matter because sample size was truly huge; where bias is a much bigger concern than parameter uncertainty - or in ignorance (either of the need for including it, or of how to do so).

[Off the top of my head the only time I've noticed it happen was when someone omitted parameter uncertainty with a forecast from a GLM, out of one of those forms of ignorance.]

It's always important to be clear about what you're predicting.

A CI for a conditional mean would consider uncertainty in the parameter value, while a PI would consider both that parameter uncertainty and observation noise. I have a couple of times seen people forget the observation noise (i.e. confusing a CI for a PI) rather than parameter uncertainty.

You use a regression example, for which the prediction interval (clearly) has two terms in the variance, and it's easy to see that they account for both the observation noise and parameter uncertainty.

I haven't read Lambert's book, but from the quote he appears to misrepresent the normal situation in frequentist forecasting.

Epistemic uncertainty is not really about uncertainty in the parameter's value, though. That's more about the model being misspecified. For example if a parameter was drifting over time, then either a Bayesian or a frequentist could misspecify that model and fail to account for that epistemic uncertainty ... or either could include it.

Similarly if it came to not really knowing what the right model might be, among a collection of plausible models, either a Bayesian or a frequentist can conduct multimodel inference. Of course one may still be wrong; perhaps the "true" situation - if there is such a thing - would still be outside the class and we may still somewhat underestimate that sort of uncertainty.

Answer 2

I would say that the Bayesian approach forces the practitioner to take the uncertainty in the parameters into account. The posterior predictive distribution automatically embeds both observation noise and parameter uncertainty. For conjugate distributions the posterior predictive can be found in closed form. In the other cases it is well defined theoretically but hard to find/compute. A large number of approaches (MCMC, HMC, etc), tools and probabilistic programming languages (STAN, pymc, pyro, etc) have been developed specifically to address this sort of questions.

In the frequentist approach the are ways to account for the uncertainty in the parameters, as in the linear regression example in the OP. However there are two issues. One is that they are only available in closed form for a limited number of cases (like linear regression) but not for most cases (even simple deviations from LR like with Huber loss or lasso regularisation). The second issue (probably related to the first one) is that in practice you'll find that in many cases (I would dare to say in most cases) practitioners actually make the mistake of fitting some parameters to some data (typically using maximum likelihood) and then use the fitted model (with point estimate of the parameters) to make predictions on new data. There are methods to make predictions with an associated interval (e.g. with "conformal predictions" techniques) but they are often add-ons, rather than part of the model like in the Bayesian approach. On the other hand, since they are less reliant on assumptions than Bayesian techniques, they may be more robust and accurate in practice.

In other words my impression is that it is virtually impossible to "forget" to account for parameters uncertainty in Bayesian statistics while it's possible (and often done "in the field") in frequentist statistics.