Timeline for What's wrong with Bonferroni adjustments?
Current License: CC BY-SA 3.0
23 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 13, 2016 at 14:11 | history | edited | amoeba | CC BY-SA 3.0 |
typo
|
| Aug 5, 2016 at 8:03 | comment | added | user83346 | @Bonferroni: I agree with you that multiplicity corrections are sometimes needed, as I explain in my answer | |
| Aug 5, 2016 at 0:27 | comment | added | Bonferroni | @FrankHarrell You seem to be confused. By quite "general principles," if you test 2 true nulls at .025, the expected number of Type I errors is...EXACTLY... E(# of errors) = P(error in test A) + P(error in test B) = .025 + .025 = .05. If you're interested in the probability of at least one error, that's where your "inequality" comes in: P(error in A or B) = P(error in A) + P(error in B) - P(error in A&B) <= .05. In any case, your distinction between inequality and equality is a red herring that does nothing to justify the implication that testing at .01 is "arbitrary" and testing at .05 isn't. | |
| Aug 4, 2016 at 22:21 | comment | added | Frank Harrell | If you think an inequality is an equality we have nothing further to discuss. If you think there are statistical principles that guide the formulation of multiplicity adjustments but cannot produce the statistical approach that starts from general principles to produce the recipe, you think differently about statistics than I do. Note that there are several inequalities besides the addition inequality that can be used if you want to remain conservative and inexact. | |
| Aug 4, 2016 at 21:57 | comment | added | Bonferroni | It's also absurd to claim that conducting each test at .01 is "arbitrary," but conducting each test at .05 is not. Even if no multiplicity adjustments are performed, one still must select the testwise alpha levels. So there is no "arbitrariness" to using equal testwise alpha levels with Bonferroni-corrected tests that doesn't exist when using equal alpha levels with uncorrected tests. In fact, even if you're just doing one test, you still have to choose the alpha level you think is appropriate. | |
| Aug 4, 2016 at 21:43 | comment | added | Bonferroni | @FrankHarrell Your other questions only serve to illustrate my point. There are often numerous choices of test statistic, test procedure, etc., even in the absence of multiplicity. That doesn't make the methodology "arbitrary" in the sense you seem to be implying. If one is interested in an omnibus test, then by all means conduct one. If one is only interested in the univariate tests, then by all means conduct the univariate tests. Are you seriously suggesting that it's "arbitrary" to select the test that addresses the question you're interested in rather than some other question? | |
| Aug 4, 2016 at 21:34 | comment | added | Bonferroni | You are equivocating two error rates. Under the null, Bonferroni EXACTLY maintains the expected number of errors per family. It gives an UPPER BOUND on the probability of "at least one" error per family (which depends on correlation). Spending alpha equally on the 5 tests is perfectly logical given no particular reason to prioritize the tests in a different way. Given another context, there are principled reasons to do otherwise. You seem to imply that it's "unprincipled" to use a mathematically sound method simply because alternative methods exist given other contexts, goals and assumptions. | |
| Aug 4, 2016 at 21:18 | comment | added | Frank Harrell | There is nothing principled about that nor is it exact in any way. Bonferroni's inequality is an upper bound for the error probability only. Why spend $\alpha$ equally on 5 parameters? Why not make an ellipsoidal region instead of a rectangular one for the acceptance region? Why not use Scheffe or Tukey's method? Why not use a simple composite ANOVA-type test? You do not achieve the desired $\alpha$ by using an inequality. | |
| Aug 4, 2016 at 21:05 | comment | added | Bonferroni | Let's say we have 5 tests and we want the expected number of Type I errors to be kept less than .05. Are you suggesting that it isn't "principled" and "non-arbitrary" to then conduct each test at the .05/5 = .01 level? That is an exact solution based on the most basic of Boolean algebra. It isn't arbitrary or unprincipled, given the prespecified goals. One selects the error rate to control (in this case, the error rate per family) and the desired alpha level (in this case, .05), and then chooses the most efficient method that satisfies those criteria. | |
| Aug 4, 2016 at 20:43 | comment | added | Bonferroni | @FrankHarrell It's not clear what you mean by "non-arbitrary." The alpha level itself (e.g. .05) is, in a sense, "arbitrary"--even if there is only a single test. And two researchers could disagree on what alpha level to use, just as they could disagree on multiple testing methodology. So it isn't the multiplicity adjustments' fault that judgments must be made. The overall alpha level, like the spending of that alpha, is chosen by the researcher based on contextual factors. Call that "arbitrary" or "ad hoc" if you want, but the same could be said of other choices, like the alpha level. | |
| Aug 4, 2016 at 12:12 | comment | added | Frank Harrell | I hope you will delve into this further. Unless you rephrase what is happening using the rules of evidence you won't really see what multiplicity adjustments are doing (although they are sometimes necessary). I also challenge you to state a principled, non-arbitrary derivation of the solution for exactly how multiplicities should be controlled for. Be sure to distinguish between controlling for overall type I error (which is not hard to do) from the recipe for exactly how one spends $\alpha$ across multiple hypothesis tests. Two statisticians applying mult. adjustments usually disagree. | |
| Aug 4, 2016 at 0:31 | comment | added | Bonferroni | @FrankHarrell I disagree. Claiming that multiplicity adjustments are "equivalent to a Bayesian prior" and "based on the underlying philosophy that the veracity of one statement depends on which other hypotheses are entertained" suggests that you fundamentally misunderstand what multiplicity adjustments are. Readers are advised that Harrell's views are not statements of mathematical fact and do not represent the dominant view among statisticians. | |
| Aug 3, 2016 at 23:02 | comment | added | Frank Harrell | You are incorrect. Although the existence of multiplicities is beyond argument, the recipe for dealing with multiplicities does not come from general frequentist principles and is always ad hoc. Secondly, it is the case that a multiplicity adjustment is equivalent to a Bayesian prior for a given parameter that becomes more skeptical as you examine other unrelated parameters. | |
| Aug 3, 2016 at 16:44 | comment | added | Bonferroni | @FrankHarrell said that multiplicity corrections "do not follow from basic statistical principles." That is definitely not true. The problem of multiplicity follows from a very basic statistical principle: the more tests, the higher the likelihood of an unusual result. Harrell also said that "multiplicity adjustments are based on the underlying philosophy that the veracity of one statement depends on which other hypotheses are entertained." That is also not true. Frequentist null hypothesis testing is concerned with rejecting the null, not with "veracity of statements." | |
| Oct 17, 2014 at 21:10 | comment | added | martino | @frank By repeatable I mean that on repeating your experiment you draw the same conclusions. In other words you can verify your results. In this way maybe researches won't be ' burned' as you put it | |
| Oct 17, 2014 at 17:34 | comment | added | goro | @ Frank Harrell. I highly appreciate what you said! The hypothesis and the correct judgment must be first and the numbers may be confirm or not and there are many factors can affect the power of the numbers | |
| Oct 17, 2014 at 17:17 | history | edited | Frank Harrell | CC BY-SA 3.0 |
elaborated on inconsistent thinking coming from multiplicity adjustments
|
| Oct 17, 2014 at 17:11 | comment | added | Frank Harrell | To answer @MJA I think there are two preferred approaches: (1) be Bayesian or (2) prioritize the hypotheses and report the results in context, in priority order. | |
| Oct 17, 2014 at 17:10 | comment | added | Frank Harrell | Not clear on the reference to 'repeatable'. If there is a single test, with no multiplicity adjustment required, the chance that a result with $P=0.04$ is repeated is not high. | |
| Oct 17, 2014 at 14:42 | comment | added | martino | Excellent point Frank. The OP is taking a classical approach though. But yes, prior knowledge should be a fundamental starting point for any statistical experiment and the the best protection against false results is that findings are repeatable – one of the cornerstones of any scientific endeavour | |
| Oct 17, 2014 at 14:04 | comment | added | goro | @ Frank Harrell. Thanks! In your opinion What is the best approach in the this case? | |
| Oct 17, 2014 at 13:59 | vote | accept | goro | ||
| Oct 17, 2014 at 15:47 | |||||
| Oct 17, 2014 at 13:32 | history | answered | Frank Harrell | CC BY-SA 3.0 |