26

What is wrong with the Bonferroni correction besides the conservatism mentioned by others is what's wrong with all multiplicity corrections. They do not follow from basic statistical principles and are arbitrary; there is no unique solution to the multiplicity problem in the frequentist world. Secondly, multiplicity adjustments are based on the underlying ...


24

The type 1 / false rejection error rate $\alpha=.05$ isn't completely arbitrary, but yes, it is close. It's somewhat preferable to $\alpha=.051$ because it's less cognitively complex (people like round numbers and multiples of five). It's a decent compromise between skepticism and practicality, though maybe a little outdated – modern methods and research ...


21

This would obviously be an absolute nightmare to do in practice, but suppose it could be done: we appoint a Statistical Sultan and everyone running a hypothesis test reports their raw $p$-values to this despot. He performs some kind of global (literally) multiple comparisons correction and replies with the corrected versions. Would this usher in a golden ...


18

How important are multiple comparisons when dealing with 6 groups? Well... with six groups you are dealing with a maximum of $\frac{6(6-1)}{2} = 15$ possible post hoc pairwise comparisons. I will let the inestimable Randall Munroe address the importance of multiple comparisons: And I will add that if, as in your opening sentence, you suggest that sometimes ...


16

Nothing went wrong. The adjusted p-values are correct. Adjusted $p=1$ simply means no evidence at all for rejecting the null hypothesis. However p.adjust(data2$raw.p, method = "holm") is always better than the Bonferroni adjustment. Holm's method, which is a step down Bonferroni adjustment, gives the same error rate control as Bonferroni but is more ...


14

The Bonferroni adjustment will always provide strong control of the family-wise error rate. This means that, whatever the nature and number of the tests, or the relationships between them, if their assumptions are met, it will ensure that the probability of having even one erroneous significant result among all tests is at most $\alpha$, your original error ...


13

It seems to me that, if a value is meaningful, its exact value is meaningful. The p value answers this question: If, in the population from which this sample was randomly drawn, the null hypothesis was true, what is the probability of getting a test statistic at least as extreme as the one we got in the sample? What about this definition makes an ...


13

With Bonferroni correction the divisor is equal to the number of tests you carry out, dependent or independent It helps to understand the purpose of the Bonferroni correction. You are testing correlations between variables. Let’s assume your null hypothesis for any two variables is that the correlation is 0. (Any null hypothesis will suffice.) Your ...


12

Fisher's test is as bad as everyone says it is from a Neyman-Pearson point of view and if you do what your question implies---after a significant ANOVA test each individual difference. You can see this in many published papers. But, testing all the differences after an ANOVA, or any of them, is neither necessary nor recommended. And, Fisher's test wasn't ...


12

He summarized saying that Bonferroni adjustment have, at best, limited applications in biomedical research and should not be used when assessing evidence about specific hypothesis. The Bonferroni correction is one of the simplest and most conservative multiple comparisons technique. It is also one of the oldest and has been improved upon greatly over time. ...


11

Generally speaking, the answer is yes, both type I and type II error rates are impacted by choosing tests on the basis of tests of assumptions. This is pretty well established with testing of equality of variance (for which several papers point it out), and testing normality. It should be expected that it will be the case in general. The advice is usually ...


11

Fisher's LSD is indeed a series of pairwise t-tests, with each test using the mean squared error from the significant ANOVA as its pooled variance estimate (and naturally taking the associated degrees of freedom). That the ANOVA be significant is an additional constraint of this test. It restricts family-wise error rate to alpha in the special case of 3 ...


10

Answer to question 1 You need to adjust for multiple comparisons if you care about the probability at which you will make a Type I error. A simple combination of metaphor/thought experiment may help: Imagine that you want to win the lottery. This lottery, strangely enough, gives you a 0.05 chance of winning (i.e. 1 in 20). M is the cost of the ticket in ...


10

Bonferroni correction is based on the fact that $$ P(\cup_{j=1}^{k} A_j) \leq \sum_{j=1}^{k} P(A_j) $$ for events $\{ A_j \}_{j=1}^{k}$, which can be a poor upper bound when the events are not disjoint. For multiple testing problems this is almost certainly the case. So in controlling the family-wise error rate by way of this bound the true error rate (...


10

Multiple comparisons corrections are intended to control the familywise error rate--or something like it--so they should be applied across a "family" of related hypothesis tests. In your first example, the overarching goal probably to determine whether Groups A and B differ. If you didn't control for multiple comparisons, you could trivially find an effect ...


9

One big distinction: The Bonferroni (or Šidák) method allows you to compute a confidence interval. The Holm method does not.


8

These two follow-up approaches have very different goals! Univariate ANOVAs (as follow-ups to MANOVA) aim at checking which individual variables (as opposed to all variables together) differ between groups. Linear Discriminant Analysis, LDA, (as a follow-up to MANOVA) aims at checking which linear combination of individual variables leads to maximal group ...


8

You would use the Bonferroni for a one-way test. But let's be clear: You would not use the Bonferroni adjustment on the Kruskal-Wallis test itself. The Kruskal-Wallis test is an omnibus test, controlling for an overall false-positive rate. You would use the Bonferroni for post hoc Dunn's pairwise tests. Indeed, Dunn introduced the "Bonferroni" adjustment. ...


8

Thanks for reading my 1997 JASA paper! If I had a do-over, I would rephrase my comment that a (single-step) Bonferroni adjusted p-value is not a probability "per se." (And I would no longer use the dreaded "per se." Yecchhhh!) The Bonferroni adjusted p is in fact an upper bound on the probability that the smallest (random) p-value is smaller than (smaller ...


8

The question is why, not what does the manual say? The answer is that alpha and the P value are different things, and they work in opposite ways. When you get a P value, you are asking "at what alpha value would I reject the null?". In other words, we are going to view our observed |t| as the critical value of the test, and the P value is the alpha ...


7

You're correct that the Holm-Bonferroni procedure is uniformly more powerful. I can see only one advantage Bonferroni has over Holm-Bonferroni. The Bonferroni correction is simple to carry out - just divide the comparison-wise error rate by k # of hypothesis tests being performed. If you're in a time crunch and need to perform a lot of hypothesis tests, ...


7

I think that you deliberately paint a pessimistic view of science produced by statistics. Indeed, in my opinion, statistics is not just a set of tools providing p values. There is also a state of rigour, care and alertness about some possible effects involved in the procedure of scientific induction... and while to my mind, everything you state is roughly ...


7

Just to add to @gordon-smyth and @student-t 's answer. Another way to look at it is to adjust the $\alpha$-level yourself instead of adjusting the $p$-values via the p.adjust() function. For the Bonferroni correction this is easy enough. If your $\alpha$ level is $0.05$, then you divide this by the number tests, which then is your new Bonferroni adjusted $\...


7

To the best of my knowledge, the Bonferroni correction is based on the number of tests you actually perform, not on the total number of pairwise tests that you could perform. So if you set out to perform two pairwise comparisons, you should divide by two.


6

The reasoning behind Fisher's LSD can be extended to cases beyond N=3. I'll discuss the case of four groups in detail. To keep the familywise Type-I error rate at 0.05 or below, a multiple-comparison correction factor of 3 (i.e. a per-comparison alpha of 0.05/3) suffices, although there are six post-hoc comparisons among the four groups. This is because: ...


6

Independence is more like a best-case assumption than a worst-case assumption. Loosely, when data are independent, each datum contains as much information as possible. If data were dependent, because their values can be predicted from other data, each additional datum must have less new information to contribute (the part that could have been predicted you ...


6

If you are correcting for Multiple testing, you need to take into account the number of tests performed. As you stated you don't need to correct for all $50*50 = 2500$ tests since $corr(X,X)=1$ and $corr(X,Y)=corr(Y,X)$. You need to correct for $(n-1)*n/2 = (49*50)/2 = 1225$ (note not $1200$ as you specified in question) tests. Note if you are in a ...


6

You absolutely do want to apply a correction. The key idea is identifying significance by chance. As you increase the number of comparisons you increase the number of those that will be significant by chance. For example, let's take the generic example of doing 100 comparisons using a significance threshold of 0.05. Now, a p-value of 0.05 means there is ...


6

You may be disappointed or relieved to hear that the answer is it depends. If your study is a preliminary exploratory study intended to determine which of the metabolites are worthy of following up, then there is no need to perform any correction for multiple comparisons. To do so robs you of power to find a real effect. On the other hand, if your experiment ...


6

Let's consider a simpler, but closely related, problem. Assume you have a vector $x$ whose entries have a correlation matrix $R$. If $R$ is diagonal, then the variance of the sample mean of $x$ is $\sigma^2_{\bar{x}} = \sum \sigma^2_i/n^2$. If all the $\sigma^2_i$ are equal, this reduces to the more commonly seen $\sigma^2/n$, and we can rearrange terms ...


Only top voted, non community-wiki answers of a minimum length are eligible