0
$\begingroup$

I have a quick question I hope somebody could help me with; If I have two one-way ANOVA's in my research just comparing the means of three age groups for a single dependent variable (so two seperate tests both for only 1 DV), and I use Bonferroni to compare the actual differences between the age groups, do I just leave the significance level at 0.05 (I think so) or do I divide by three because of the age groups? I thought I could leave it, but I know this Bonferroni correction of the significance level exists with multiple tests so I just want to check whether that is the case here?

Kind regards :)

Improve this question
$\endgroup$
2
  • $\begingroup$ If you don’t divide by the number of comparisons, you’re not doing a Bonferroni correction, so you would test at $\alpha=0.05/3$. Bonferroni, however, is conservative and quickly sucks away your power to reject. Consider another method. The p.adjust command in R has some options. $\endgroup$
    – Dave
    Feb 11, 2020 at 13:01
  • $\begingroup$ Thanks @Dave, I see your point. If I am not dividing the significance for my three comparisons, I am not really doing a post-hoc analysis at all xd. Thanks and I will do it! $\endgroup$
    – CalcBoy
    Feb 11, 2020 at 13:14

2 Answers 2

Reset to default
1
$\begingroup$

In the words of Jacob Cohen the question of whether, when and how to correct for multiple corrections is one where "intelligent people disagree".

One problem with the corrections is that, by lowering type 1 error they increase type 2 error.

Another is how much correction to do. What is the number of comparisons? Everything in one paper? One table? One hypothesis? Every analysis you do in your career?

Improve this answer
$\endgroup$
-3
$\begingroup$

Multiple testing corrections are used when testing a large number of individual hypothesis and it would not be admissible to have a large number of errors. If I am performing 10,000 tests and keep a significance level of 0.05(5%), I will obtain 500 type-1 errors (1 in every 20). The significance levels (often referred to as $\alpha$) is equivalent to the probability of the type-1 error (i.e., saying something is significant when it actually is not). When performing a large number of tests, we can use the Bonferroni correction (amongst many other methods) to update our significance value to $\frac{0.05}{number of tests}$ (e.g. 0.05/10000 = $5\cdot 10^{-6}$).

In conclusion, in your case (where you are performing 3 tests), you do not need to do Bonferroni correction.

You can read more here

Improve this answer
$\endgroup$
3
  • $\begingroup$ -1 if you make no correction for the three tests, at $\alpha=0.05$, your probability of a type I error is almost 15%! $1-(0.95)^3$ $\endgroup$
    – Dave
    Feb 11, 2020 at 13:07
  • 1
    $\begingroup$ Sorry, but this is not correct. Whether you need to use Bonferroni (or any correction) is debatable. But if you need it for 10,000 tests then you need it for 3. $\endgroup$
    – Peter Flom
    Feb 11, 2020 at 13:12
  • $\begingroup$ Apologies for my mistake. I will take note of this. Still, the use of Bonferroni is debatable for a small number of tests as the reduction may result in an increase of type II errors. The use of it pretty much depends on how acceptable it is to get type I or type II errors. $\endgroup$
    – Luis Chaves Rodriguez
    Feb 12, 2020 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.