22
$\begingroup$

Andrew Gelman wrote an extensive article on why Bayesian AB testing doesn't require multiple hypothesis correction: Why We (Usually) Don’t Have to Worry About Multiple Comparisons, 2012.

I don't quite understand: why don't Bayesian methods require multiple testing corrections?

A ~ Distribution1 + Common Distribution
B ~ Distribution2 + Common Distribution
C ~ Distribution3 + Common Distribution
Common Distribution ~ Normal

My understanding is that the Bayesian approach shown above accounts for the shared underlying distribution by all the hypothesis (unlike in a frequentist Bonferroni correction). Is my reasoning correct?

$\endgroup$
14
$\begingroup$

One odd way to answer the question is to note that the Bayesian method provides no way to do this because Bayesian methods are consistent with accepted rules of evidence and frequentist methods are often at odds with them. Examples:

  • With frequentist statistics, comparing treatment A to B must penalize for comparing treatments C and D because of family-wise type I error considerations; with Bayesian the A-B comparison stands on its own.
  • For sequential frequentist testing, penalties are usually required for multiple looks at the data. In a group sequential setting, an early comparison for A vs B must be penalized for a later comparison that has not been made yet, and a later comparison must be penalized for an earlier comparison even if the earlier comparison did not alter the course of the study.

The problem stems from the frequentist's reversal of the flow of time and information, making frequentists have to consider what could have happened instead of what did happen. In contrast, Bayesian assessments anchor all assessment to the prior distribution, which calibrates evidence. For example, the prior distribution for the A-B difference calibrates all future assessments of A-B and does not have to consider C-D.

With sequential testing, there is great confusion about how to adjust point estimates when an experiment is terminated early using frequentist inference. In the Bayesian world, the prior "pulls back" on any point estimates, and the updated posterior distribution applies to inference at any time and requires no complex sample space considerations.

$\endgroup$
  • 4
    $\begingroup$ I don't really understand this argument. If we conduct 1000 different comparisons with a usual frequentist approach then of course we should expect around 50 significant with p<0.05 effects even under the null. Hence the corrections. If we use Bayesian estimation/testing instead, having some prior (around 0?) for all comparisons, then yes the prior will shrink the posteriors toward zero, but we would still have randomly varying posteriors and/or Bayes factors and will probably have some cases out of 1000 that will look like "substantial" effects, even when true effects are all zero. $\endgroup$ – amoeba Mar 24 '16 at 13:23
  • 1
    $\begingroup$ @amoeba - one way to consider it is that Bayesian takes account of all alternatives - not just "null" vs "one alternative". Considering all alternative means generally each one has smaller prior probability - effectively penalizing the inference. You have to consider all $2^{1000} $ combinations of true/false (assuming you have no prior knowledge of combinations that are impossible). You are worried about something going wrong in *just one case*. What about the other $2^{1000}-1$ cases? $\endgroup$ – probabilityislogic Mar 24 '16 at 13:43
  • 1
    $\begingroup$ Sorry, @probabilityislogic, I am not sure I understood your point. Fair enough about "all alternatives", but what happens in practice? As I said, we are estimating 1000 group differences (for example); we have a prior on group difference; we obtain 1000 posteriors, 95% credible intervals, or whatever. Then we'd look at each credible interval to check if it's far enough from zero to be a "meaningful/substantial" effect. If we do this 1000 times, we are likely to have some "false positives" in a sense that some effects will appear large even if all 1000 effects are in fact equal to zero. No? $\endgroup$ – amoeba Mar 24 '16 at 14:07
  • 1
    $\begingroup$ @amoeba - your argument depends on those $1000$ intervals/rejections being independent. In practice, people usually don't test large numbers of unrelated hypothesis. Hence the multilevel model - to capture the common influences. This will make those credible intervals move together (ie they will have correlated sampling distributions). This will lead to more false positives when bad models are used, and less when good models are used. Of course, good or bad is in terms of having enough information incorporated into the models. $\endgroup$ – probabilityislogic Mar 24 '16 at 16:12
  • 1
    $\begingroup$ @probabilityislogic: Well, I am absolutely in favour of multilevel models, even though I don't see them necessarily as a Bayesian tool -- mixed models and ANOVAs with random effects are commonly used alongside with t-tests and such... $\endgroup$ – amoeba Mar 24 '16 at 17:22
6
$\begingroup$

This type of hierarchical model does shrink the estimates and reduces the number of false claims to a reasonable extent for a small to moderate number of hypotheses. Does it guarantee some specific type I error rate? No.

This particular suggestion by Gelman (who acknowledges the issue with looking at too many different things and then too easily wrongly concluding that you see something for some of them - in fact one of his pet topics on his blog) is distinct from a the extreme alternative viewpoint that holds that Bayesian methods do not need to account for multiplicity, because all that matters are your likelihood (and your prior).

$\endgroup$
6
$\begingroup$

Very interesting question, here's my take on it.

It's all about encoding information, then turn the Bayesian crank. It seems too good to be true - but both of these are harder than they seem.

I start with asking the question

What information is being used when we worry about multiple comparisons?

I can think of some - the first is "data dredging" - test "everything" until you get enough passes/fails (I would think almost every stats trained person would be exposed to this problem). You also have less sinister, but essentially the same "I have so many tests to run - surely all can't be correct".

After thinking about this, one thing I notice is that you don't tend to hear much about specific hypothesis or specific comparisons. It's all about the "collection" - this triggers my thinking towards exchangeability - the hypothesis being compared are "similar" to each other in some way. And how do you encode exchangeability into bayesian analysis? - hyper-priors, mixed models, random effects, etc!!!

But exchangeability only gets you part of the way there. Is everything exchangeable? Or do you have "sparsity" - such as only a few non-zero regression coefficients with a large pool of candidates. Mixed models and normally distributed random effects don't work here. They get "stuck" in between squashing noise and leaving signals untouched (e.g. in your example keep locationB and locationC "true" parameters equal, and set locationA "true" parameter arbitrarily large or small, and watch the standard linear mixed model fail.). But it can be fixed - e.g. with "spike and slab" priors or "horse shoe" priors.

So it's really more about describing what sort of hypothesis you are talking about and getting as many known features reflected in the prior and likelihood. Andrew Gelman's approach is just a way to handle a broad class of multiple comparisons implicitly. Just like least squares and normal distributions tend to work well in most cases (but not all).

In terms of how it does this, you could think of a person reasoning as follows - group A and group B might have the same mean - I looked at the data, and the means are "close" - Hence, to get a better estimate for both, I should pool the data, as my initial thought was they have the same mean. - If they are not the same, the data provides evidence that they are "close", so pooling "a little bit" won't hurt me too badly if my hypothesis was wrong (a la all models are wrong, some are useful)

Note that all the above hinges on the initial premise "they might be the same". Take that away, and there is no justification for pooling. You can probably also see a "normalish distribution" way of thinking about the tests. "Zero is most likely", "if not zero, then close to zero is next most likely", "extreme values are unlikely". Consider this alternative:

  • group A and group B means might be equal, but they could also be drastically different

Then the argument about pooling "a little bit" is a very bad idea. You are better off choosing total pooling or zero pooling. Much more like a Cauchy, spike&slab, type of situation (lots of mass around zero, and lots of mass for extreme values)

The whole multiple comparisons doesn't need to be dealt with, because the Bayesian approach is incorporating the information that leads us to worry into the prior and/or likelihood. In a sense it more a reminder to properly think about what information is available to you, and making sure you have included it in your analysis.

$\endgroup$
  • 2
    $\begingroup$ So one way to squish coefficients to be zeroes unless there's really something going on is lasso; in the frequentist version of it, you apply an $l_1$ norm on the sum of the coefficients, and in the Bayesian version of it, you use sharply peaked priors (Laplace $\exp(-|x|)$). So in this case, knowing that you want to have mostly zeroes and a handful of nonzeroes in your output, you modify the prior to correspond to that statement that zero is a much more likely value than any other (vs. the normal prior's statement that values near zero are about as likely as the zero itself). $\endgroup$ – StasK Mar 24 '16 at 15:02
  • $\begingroup$ @StasK - l1 would work better, but as it is log-concave would struggle with sparse non-zeros. The ones I mentioned are all log-convex. A close variant to l1 is generalised double pareto - get by taking a mixture of laplace scale parameter (similar to adaptive lasso in ML speak) $\endgroup$ – probabilityislogic Mar 24 '16 at 15:10
5
$\begingroup$

First, as I understand the model you presented I think it is a bit different to Gelman proposal, that more looks like:

A ~ Distribution(locationA)
B ~ Distribution(locationB)
C ~ Distribution(locationC)

locationA ~ Normal(commonLocation)
locationB ~ Normal(commonLocation)
locationC ~ Normal(commonLocation)

commonLocation ~ hyperPrior

In practice, by adding this commonLocation parameter, the inferences over the parameters the 3 distributions (here locations 1, 2 and 3) are no longer independent from each other. Moreover, commonLocation tends to shrink expectational values of the parameters toward a central (generally estimated) one. In a certain sense, it works as a regularization over all the inferences making the need of correction for multiple correction not needed (as in practice we perform one single multivariate estimation accounting from the interaction between each of them through the use of model).

As pointed out by the other answer, this correction does not offer any control on type I error but in most cases, Bayesian method does not offer any such control even at the single inference scale and correction for multiple comparison must be thought differently in the Bayesian setting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy