2
$\begingroup$

Are Bayesian models subject to the same problems as frequentist ones, where we cannot run a bunch of different models due to Type I error? For example, let's say I have a large data frame on airplanes, dependent variable miles flown, independent variable total passengers, and a grouping variable of airline company (made-up example). For frequentist statistics, I really shouldn't split the data frame into a bunch of smaller ones based on airline company and run a different regression for each company, I should run a single regression with company as an included variable (otherwise, I get major increases in Type I error). Do the same rules apply if I approach the same problem from a Bayesian perspective?

Since Bayesian approaches are more calculation-heavy and take forever on larger datasets, it would be helpful to be able to split the data frame and run multiple models, combining the results from the CIs at the end. But I'm new to Bayesian stats, so I'm not sure if this is a bad thing to do.

$\endgroup$

1 Answer 1

9
$\begingroup$

Yes, looking at multiple questions and deciding "reject one or more null hypotheses" based on a Bayesian analysis (e.g. based on some cut-off for a Bayes factor, or if posterior credible intervals for some parameters exclude the values consistent with the null hypothesis), does result in a similar type I error inflation situation just the same as if you were using a frequentist approach.

There's two things to point out though:

  1. Conservative priors and/or hierarchical shrinkage of parameters that you do tests about can (in some situations) make the type I error rate less badly inflated by multiple testing than a traditional frequentist approach might. However, it's hard to know in general how much this helps, while for any particular set-up one can explore this through e.g. simulations.
  2. Some would argue that type I error control is just not a relevant concept ("sure the type I error rate is inflated, but we don't care about type I error rate in the first place" - sometimes people will add that they don't believe any point null hypothesis is exactly true in the first place anyway), while others will argue that frequentist operating characteristics are how one should evaluate Bayesian methods.
$\endgroup$
2
  • 1
    $\begingroup$ As a supporting point to the answer that has already been provided, you could be interested in reading the following paper by Gelman "Why we (usually) don’t have to worry about multiple comparisons", where the authors propose that the problem of multiple comparisons can disappear entirely when viewed from a hierarchical Bayesian perspective. * I have answered since I don't have enough points to comment yet. I would be ok in deleting if this is deemed as not useful or if anyone else wants to comment my response on the mai $\endgroup$
    – Javier
    Dec 6, 2023 at 23:35
  • $\begingroup$ Indeed, was thinking of that paper for (1). More nuanced discussions: here & here incl. limitations (e.g. Bayesian inference with wide priors/exactly right priors that you made a little wider "to be on the conservative side" etc. all don't enjoy the optimality properties of the correct Bayesian model with exactly right priors...). $\endgroup$
    – Björn
    Dec 7, 2023 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.