# Correcting for multiple comparisons after multiple ANOVAs

I've had to run 5 different ANOVAs (identical 4-way mixed design ANOVAs including the same factors, but on a different dependent variable [a gait parameter] each time). In this case, what would be the best way to correct my p-values? Let's say I'm using Bonferroni (for example)... am I dividing my alpha by 5 since that's how many ANOVAs I've done?

There are other analyses that have been done. For example an additional ANOVA was run to compare walking pace in various conditions to the target walking pace (again, a new dependent variable but still including the factors from the previous ANOVAs). So I suppose I should be correcting for 6 tests. Then, of course, to tease apart interactions in the various ANOVAs I have t-tests... do they come into the overall adjustment?

I think I'm getting lost in all the numbers, and would really appreciate some input on how to handle multiple comparison corrections in this situation. I'm not certain how clear I am, so I can provide more detail if needed.

• It looks like you will be testing 90 effects. Can you consider some primary and others secondary? Also, have you considered a MANOVA? – David Lane Jan 30 '17 at 5:35
• Hi David. I did consider a MANOVA but didn't have a solid balance of spatial vs temporal parameters of gait to feel it was a good canonical variable. I also had a covariate in one of my ANOVAs, which seemed to complicate things. Although my committee urged me to move forward with the multiple ANOVAs I would like to revisit the possibility of a MANOVA at some point. But unfortunately that's not what I'm working with here. – eread Jan 31 '17 at 0:45
• Hi eread, your fine with ANOVA's. Keep in mind that although one reason to do a MANOVA is to explore the canonical structure, another is to simply control the Type I error rate. – David Lane Jan 31 '17 at 0:59
• Thanks David. So I think where I've gotten lost is between what I said above about dividing the alpha and what you've said about me testing 90 effect. I did use multiple comparison corrections in my pairwise comparisons already, but I'm feeling lost about what the appropriate value is to apply in my Bonferroni. Hope you don't mind the follow up question - I seem to have gotten into a vortex of stats reading and am appreciating the external input. – eread Jan 31 '17 at 1:52
• Hi eread, The number 90 comes 15 effects per ANOVA x 6 DV's. This is a question without a "correct" answer. You have to make a tradeoff between protecting the the Type I error rate and power. The most conservative thing would be to use the .05/90 significance level but that would have very low power. If you could choose a smaller set of tests that you care most about and divide by that number instead of 90. – David Lane Jan 31 '17 at 3:53

Not to worry too much. Now, are the results being interpreted serially or not, i.e., is this or that or the next thing or the next etc. significant? If, for example, we have 6 tests and they are all significant, then we are not saying that a single one of them determines that the whole series of tests is significant, it just is not that situation.

So what is Bonferroni? Bland-Altman explain it thus "If we test a null hypothesis which is in fact true, using 0.05 as the critical significance level, we have a probability of 0.95 of coming to a not significant—that is, correct—conclusion. If we test two independent true null hypotheses, the probability that neither test will be significant is 0.95x0.95=0.90. If we test 20 such hypotheses the probability that none will be significant is $0.95^{20}=0.36$. This gives a probability of 1–0.36=0.64 of getting at least one significant result—we are more likely to get one than not. The expected number of spurious significant results is 20x0.05=1. In general, if we have ($\kappa$) independent significant tests at the ($\alpha$) level of null hypotheses which are all true, the probability that we will get no significant differences is $(1-\alpha)^{\kappa}$. If we make ($\alpha$) small enough we can make the probability that none of the separate tests is significant equal to 0.95. Then if any of the ($\kappa$) tests has a $P$-value less than ($\alpha$) we will have a significant difference between the treatments at the 0.05 level. Since $\alpha$ will be very small, it can be shown that $(1-\alpha)^{\kappa} \approx 1-\kappa \alpha$. If we put $\kappa \alpha=0.05$, so $\alpha=\frac{0.05}{\kappa}$, we will have probability 0.05 that one of the $\kappa$ tests will have a $P$ value less than $\alpha$ if the null hypotheses are true. Thus, if in a clinical trial we compare two treatments within five subsets of patients the treatments will be significantly different at the 0.05 level if there is a P value less than 0.01 within any of the subsets. This is the Bonferroni method. Note that they are not significant at the 0.01 level, but at only the 0.05 level."

On the other hand, if your situation is a MANOVA one, as @DavidLane suggests, then you should use that first, as it will lump all the data into a single test of significance and be more specific to the lumped significance than a Bonferroni correction of multiple serial tests. Your question "to tease apart interactions in the various ANOVAs I have t-tests... do they come into the overall adjustment?" That could be done after MANOVA to first see if the ensemble is significant.

• You have some math notation issues here, e.g. "0.9520=0.36" and "(1-(alpha))(kappa)(about)1-(kappa) (alpha)". Also, what about the part where the OP says "Then, of course, to tease apart interactions in the various ANOVAs I have t-tests... do they come into the overall adjustment?"? Also, I don't understand the second sentence of your first paragraph. Also, this appears to be a block quote from another paper; maybe you should paraphrase....Just trying to prevent the conniption that will ensue after the inevitable serial downvotes of this answer. – gammer Jan 30 '17 at 5:45
• @gammer Yeah, I noticed that, and I fixed that, it just took a while. – Carl Jan 30 '17 at 5:58
• Thanks your answer, Carl. If I'm understanding your question properly, I don't think they were serial. Interpreting one ANOVA wasn't dependent on the significance of a previous one. Is that what you meant? I considered running a MANOVA but I didn't have a good enough balance between spatial parameters and temporal parameters. I also had to use a covariate in one of the ANOVAs which complicated it. I know a MONOVA would be ideal since it provides the "protected F factor". I should perhaps revisit this, though my committee initially said to move forward with several ANOVAs. – eread Jan 31 '17 at 0:42
• @eread Well, actually, yes, I do mean serial, but that would not be a common way to explain the problem. However, suppose you have a machine composed 10 serial processes (like gears boxes, motors, pumps, etc.) and the output of each process feeds material into the next input. If each process works 95% of the time, and the processes break down independently of each other: a jammed gear, a burnt coil, a leaking pump that loses pressure, then the entire machine works only $0.95^{10}\approx 0.599$ or just under 60% of the time. That situation is the same as the Bonferroni argument. – Carl Jan 31 '17 at 2:05