4
$\begingroup$

As the title: Do Bayes factors require mutliple comparion correction?

For more context, I am calculating very many likelihood ratio tests and I am thinking about how to handle multiple comparison correction. I thought Bayes factors might present a solution - if I am presenting the results in the Bayes factor evidence scale then I think correction should not be required?

Computing full Bayes factors for every test with MCMC would be difficult (but not impossible - although I am not sure how to chose priors really), but following Wasserman (2000), it seems that BIC can be used to approximate the Bayes factor. So it seems to get around my multiple comparisons difficulties I can simply add the $\frac{d_f}{2}\log n$ term to my log-likelihood ratio, exponentiate it and call it a Bayes factor which I can present without correction.

It seems too good to be true, so what am I missing?

My view is that it pushes the inference step to the reader instead presenting the Bayes factor evidence (which is of course exactly what I want to do, philosophically I don't see the need for assigning every point in an image a precise p-value) - but is this likely to be acceptable to reviewers? (The application is in neuroimaging)

$\endgroup$
3
$\begingroup$

What you are missing is that it very rarely makes sense to use a prior for which all of parameters are independent. That might make sense if the parameters were as varied and logically disconnected as, say, the set {some physical constant, some baseball player's batting average, some yeast gene's expression level}.

Most analyses look at sets of parameters that are best modeled as exchangeable, like, say, the set of all current players' batting averages or the set of all yeast genes' expression levels. For estimation, this leads to approaches like this; for testing, it leads to approaches like this.

$\endgroup$
2
$\begingroup$

In the context of frequentist inference, the posterior odds, or posterior probability of the null (a.k.a. local fdr), is nothing more than a test statistic. It can definitely lead you to many erroneous inferences due to chance alone.

If you want some frequentist guarantee on the amount of errors you might be making, the posterior odds should be plugged into some multiplicity control scheme.

$\endgroup$
  • $\begingroup$ Thanks. I understand for the frequentist interpretation correction is necessary. But do Bayes factors with a bayesian interpretation require any correction? $\endgroup$ – thrope Apr 23 '13 at 8:52
  • $\begingroup$ I would say no. But have a look at this:normaldeviate.wordpress.com/2012/11/09/anti-xkcd $\endgroup$ – JohnRos Apr 23 '13 at 11:36
2
$\begingroup$

Not necessarily for Bayes factors. But Bayes factors require conversion to posterior probabilities for proper inference, and posterior probabilities most definitely can require a Bonferroni-style multiplicity adjustment of sorts. Here is the explanation:

If the hypotheses are independent a priori, then probability that all nulls are simultaneously true decreases to zero very rapidly as the number of hypotheses tested increases. But there may be scientific doubt as to whether any effect exists: For example, in a neuroimaging study there can be doubt as to whether the association studied has any neuronal basis whatsoever. (Nevertheless, the most extreme statistics are likely to suggest an association, simply because when you look at a large number of statistics, the extreme values will likely be quite atypical.)

In cases such as this where there is scientific doubt about whether there is any association whatsoever, you must set your prior probability on the global null hypothesis to some non-infinitesimal value in order to correctly model such doubt. In doing so, your prior probabilities on the component nulls are necessarily increased from commonly-used levels (such as 0.5), depending on the number of hypotheses tested, and depending on your assessment of their degree of prior dependence. Such a change in the prior probabilities on the component nulls causes a change (or adjustment) in the posterior probabilities on the component nulls. This adjustment is similar to the usual Bonferroni adjustment when the components are assumed independent a priori, and is less extreme than the Bonferroni adjustment when the components are assumed dependent.

This issue is discussed in the following literature. The last reference discusses how to do the analysis when you assume prior dependence.

Jeffreys, H. Theory of Probability. Oxford press.

Westfall, P.H., Johnson, W.O. and Utts, J.M.(1997). A Bayesian Perspective on the Bonferroni Adjustment, Biometrika 84, 419–427.

Gönen, M., Westfall, P.H. and Johnson, W.O. (2003). Bayesian multiple testing for two-sample multivariate endpoints, Biometrics 59, 76–82.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.