Calculating number of tests in Bonferroni Correction I understand that applying Bonferroni correction involves dividing alpha by number of tests performed. However, I am unsure about how I can determine the number of tests to divide by.
I have 6 IVs that are entered into a simultaneous equation to predict a DV. For exploratory analysis, the DV is further split into 3 sub-scores, each measured across 4 time points. The 6 IVs are then measured for each score at each time point. Does this means that I would need to divide alpha by 12 (3 scores * 4 time points), 6 (6 IVs) or 72?
Thanks in advance!
 A: Like for many things in statistics there is no fixed rule about how the Bonferroni correction is to be applied. The Bonferroni correction is meant to achieve a certain aim. If you have $k$ tests and a level of $\alpha$, and you apply the Bonferroni correction over these $k$ tests, it means that assuming all $k$ null hypotheses to be true, the probability of finding at least one significant rejection in your $k$ tests is smaller or equal than $\alpha$.
So in your situation, if you divide by 72, the probability to find any wrong significance is smaller or equal than $\alpha$. This is the safest thing you can do, but it has the obvious disadvantage that also the power, i.e., the probability to find a "correct" significance in case that a null hypothesis is false, is lower than if you use a lower $k$ for the Bonferroni correction. Some people would divide by 6 for the IVs (so that their probability for finding a wrong significance among those tests only is $\le\alpha$), and then by 12 for each of the sub-score/time point-tests, to secure the $\alpha$ for each group of them in isolation, which has a better power than dividing by 72, but less protection against wrong significances.
The approach that I would probably take is to report results at various levels, like "tests A and B are significant at level $\alpha/72$, which secures an overall level $\alpha$ by means of Bonferroni. A number of further tests still give an indication that something may go on, which would deserve further investigation. Test C and D are significant at $\alpha/6$ (Bonferroni applied to IVs only), and test E and F are just significant at level $\alpha$, meaning that these could easily be meaningless, however they give some weak indication against their null hypotheses."
