34
$\begingroup$

Andrew Gelman wrote an extensive article on why Bayesian AB testing doesn't require multiple hypothesis correction: Why We (Usually) Don’t Have to Worry About Multiple Comparisons, 2012.

I don't quite understand: why don't Bayesian methods require multiple testing corrections?

A ~ Distribution1 + Common Distribution
B ~ Distribution2 + Common Distribution
C ~ Distribution3 + Common Distribution
Common Distribution ~ Normal

My understanding is that the Bayesian approach shown above accounts for the shared underlying distribution by all the hypothesis (unlike in a frequentist Bonferroni correction). Is my reasoning correct?

Improve this question
$\endgroup$
3
  • $\begingroup$ I would follow Andrew Gelman's example: Why We (Usually) Don’t Have to Worry About Multiple Comparisons, 2012. See also on his blog. $\endgroup$
    – Patrick McCann
    Commented Oct 17, 2011 at 18:29
  • 6
    $\begingroup$ While Patrick's links are very helpful, it would be great to see a more self-contained answer geared to the 'moderately statistically literate scientific reviewer'. $\endgroup$
    – conjectures
    Commented Oct 9, 2013 at 11:52
  • $\begingroup$ I propose that the question to ask is: For any quantity you will estimate from the posterior distribution, is the prior distribution uninformative? This provides a fairly mechanical way to criticize Bayesian priors. Quantities might be things like "what is the largest difference in means between groups?" or "how many groups differ from each other by a non-trivial amount?" -- if you have many groups it's easy to accidentally have a far from uninformative prior on these questions. $\endgroup$
    – Paul Harrison
    Commented Jun 30, 2022 at 3:04

4 Answers 4

Reset to default
17
$\begingroup$

One odd way to answer the question is to note that the Bayesian method provides no way to do this because Bayesian methods are consistent with accepted rules of evidence and frequentist methods are often at odds with them. Examples:

  • With frequentist statistics, comparing treatment A to B must penalize for comparing treatments C and D because of family-wise type I error considerations; with Bayesian the A-B comparison stands on its own.
  • For sequential frequentist testing, penalties are usually required for multiple looks at the data. In a group sequential setting, an early comparison for A vs B must be penalized for a later comparison that has not been made yet, and a later comparison must be penalized for an earlier comparison even if the earlier comparison did not alter the course of the study.

The problem stems from the frequentist's reversal of the flow of time and information, making frequentists have to consider what could have happened instead of what did happen. In contrast, Bayesian assessments anchor all assessment to the prior distribution, which calibrates evidence. For example, the prior distribution for the A-B difference calibrates all future assessments of A-B and does not have to consider C-D.

With sequential testing, there is great confusion about how to adjust point estimates when an experiment is terminated early using frequentist inference. In the Bayesian world, the prior "pulls back" on any point estimates, and the updated posterior distribution applies to inference at any time and requires no complex sample space considerations.

Improve this answer
$\endgroup$
10
  • 10
    $\begingroup$ I don't really understand this argument. If we conduct 1000 different comparisons with a usual frequentist approach then of course we should expect around 50 significant with p<0.05 effects even under the null. Hence the corrections. If we use Bayesian estimation/testing instead, having some prior (around 0?) for all comparisons, then yes the prior will shrink the posteriors toward zero, but we would still have randomly varying posteriors and/or Bayes factors and will probably have some cases out of 1000 that will look like "substantial" effects, even when true effects are all zero. $\endgroup$
    – amoeba
    Commented Mar 24, 2016 at 13:23
  • 4
    $\begingroup$ Sorry, @probabilityislogic, I am not sure I understood your point. Fair enough about "all alternatives", but what happens in practice? As I said, we are estimating 1000 group differences (for example); we have a prior on group difference; we obtain 1000 posteriors, 95% credible intervals, or whatever. Then we'd look at each credible interval to check if it's far enough from zero to be a "meaningful/substantial" effect. If we do this 1000 times, we are likely to have some "false positives" in a sense that some effects will appear large even if all 1000 effects are in fact equal to zero. No? $\endgroup$
    – amoeba
    Commented Mar 24, 2016 at 14:07
  • 3
    $\begingroup$ @probabilityislogic: Well, I am absolutely in favour of multilevel models, even though I don't see them necessarily as a Bayesian tool -- mixed models and ANOVAs with random effects are commonly used alongside with t-tests and such... $\endgroup$
    – amoeba
    Commented Mar 24, 2016 at 17:22
  • 4
    $\begingroup$ @amoeba the problem with the reasoning is that you are envisioning events that Bayesians (and most non-statisticians) are not interested in. I am not interested in computing the probability that at least one of 1000 experiments will give a misleading result. I want to know the evidence for an effect based on exactly one experiment, i.e., the posterior probability of an effect given the available data. $\endgroup$
    – Frank Harrell
    Commented Mar 24, 2016 at 17:50
  • 3
    $\begingroup$ I go farther than that. Bayesians are not needing multiplicity corrections because rules of evidence need to deal with the veracity of one-at-a-time statements and not a universe of assertions. The events Bayesians are interested in calculating the probabilities of are not involving theoretical experiment repetitions. $\endgroup$
    – Frank Harrell
    Commented Mar 24, 2016 at 19:42
7
$\begingroup$

This type of hierarchical model does shrink the estimates and reduces the number of false claims to a reasonable extent for a small to moderate number of hypotheses. Does it guarantee some specific type I error rate? No.

This particular suggestion by Gelman (who acknowledges the issue with looking at too many different things and then too easily wrongly concluding that you see something for some of them - in fact one of his pet topics on his blog) is distinct from a the extreme alternative viewpoint that holds that Bayesian methods do not need to account for multiplicity, because all that matters are your likelihood (and your prior).

Improve this answer
$\endgroup$
2
7
$\begingroup$

First, as I understand the model you presented I think it is a bit different to Gelman proposal, that more looks like:

A ~ Distribution(locationA)
B ~ Distribution(locationB)
C ~ Distribution(locationC)

locationA ~ Normal(commonLocation)
locationB ~ Normal(commonLocation)
locationC ~ Normal(commonLocation)

commonLocation ~ hyperPrior

In practice, by adding this commonLocation parameter, the inferences over the parameters the 3 distributions (here locations 1, 2 and 3) are no longer independent from each other. Moreover, commonLocation tends to shrink expectational values of the parameters toward a central (generally estimated) one. In a certain sense, it works as a regularization over all the inferences making the need of correction for multiple correction not needed (as in practice we perform one single multivariate estimation accounting from the interaction between each of them through the use of model).

As pointed out by the other answer, this correction does not offer any control on type I error but in most cases, Bayesian method does not offer any such control even at the single inference scale and correction for multiple comparison must be thought differently in the Bayesian setting.

Improve this answer
$\endgroup$
0
6
$\begingroup$

Very interesting question, here's my take on it.

It's all about encoding information, then turn the Bayesian crank. It seems too good to be true - but both of these are harder than they seem.

I start with asking the question

What information is being used when we worry about multiple comparisons?

I can think of some - the first is "data dredging" - test "everything" until you get enough passes/fails (I would think almost every stats trained person would be exposed to this problem). You also have less sinister, but essentially the same "I have so many tests to run - surely all can't be correct".

After thinking about this, one thing I notice is that you don't tend to hear much about specific hypothesis or specific comparisons. It's all about the "collection" - this triggers my thinking towards exchangeability - the hypothesis being compared are "similar" to each other in some way. And how do you encode exchangeability into bayesian analysis? - hyper-priors, mixed models, random effects, etc!!!

But exchangeability only gets you part of the way there. Is everything exchangeable? Or do you have "sparsity" - such as only a few non-zero regression coefficients with a large pool of candidates. Mixed models and normally distributed random effects don't work here. They get "stuck" in between squashing noise and leaving signals untouched (e.g. in your example keep locationB and locationC "true" parameters equal, and set locationA "true" parameter arbitrarily large or small, and watch the standard linear mixed model fail.). But it can be fixed - e.g. with "spike and slab" priors or "horse shoe" priors.

So it's really more about describing what sort of hypothesis you are talking about and getting as many known features reflected in the prior and likelihood. Andrew Gelman's approach is just a way to handle a broad class of multiple comparisons implicitly. Just like least squares and normal distributions tend to work well in most cases (but not all).

In terms of how it does this, you could think of a person reasoning as follows - group A and group B might have the same mean - I looked at the data, and the means are "close" - Hence, to get a better estimate for both, I should pool the data, as my initial thought was they have the same mean. - If they are not the same, the data provides evidence that they are "close", so pooling "a little bit" won't hurt me too badly if my hypothesis was wrong (a la all models are wrong, some are useful)

Note that all the above hinges on the initial premise "they might be the same". Take that away, and there is no justification for pooling. You can probably also see a "normalish distribution" way of thinking about the tests. "Zero is most likely", "if not zero, then close to zero is next most likely", "extreme values are unlikely". Consider this alternative:

  • group A and group B means might be equal, but they could also be drastically different

Then the argument about pooling "a little bit" is a very bad idea. You are better off choosing total pooling or zero pooling. Much more like a Cauchy, spike&slab, type of situation (lots of mass around zero, and lots of mass for extreme values)

The whole multiple comparisons doesn't need to be dealt with, because the Bayesian approach is incorporating the information that leads us to worry into the prior and/or likelihood. In a sense it more a reminder to properly think about what information is available to you, and making sure you have included it in your analysis.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ So one way to squish coefficients to be zeroes unless there's really something going on is lasso; in the frequentist version of it, you apply an $l_1$ norm on the sum of the coefficients, and in the Bayesian version of it, you use sharply peaked priors (Laplace $\exp(-|x|)$). So in this case, knowing that you want to have mostly zeroes and a handful of nonzeroes in your output, you modify the prior to correspond to that statement that zero is a much more likely value than any other (vs. the normal prior's statement that values near zero are about as likely as the zero itself). $\endgroup$
    – StasK
    Commented Mar 24, 2016 at 15:02
  • $\begingroup$ @StasK - l1 would work better, but as it is log-concave would struggle with sparse non-zeros. The ones I mentioned are all log-convex. A close variant to l1 is generalised double pareto - get by taking a mixture of laplace scale parameter (similar to adaptive lasso in ML speak) $\endgroup$
    – probabilityislogic
    Commented Mar 24, 2016 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.