Correction for multiple comparisons I have a 2x3x4 repeated measured anova. I have a significant 3-way interaction, and I want to make sure that I am using the correct post hoc comparisons and not violating any key statistics theory.
I have run the statistics in SPSS and have adjusted for multiple comparisons using SIDAK but I want to make sure I understand how many adjustments are being made and if the p values are being corrected appropriately.
My post hoc tests have analysed the following:
AxB at each level of C
AxC at each level of B
BXC at each level of A
I am just trying to determine what correction factor is appropriate. For example b x c at each level of A compares 2 means 12 times. Here am I not adjusting because it is 2 means, or I am dividing the alpha value by 12 as there are 12 comparisons.
Similarly for a x c at each level of B I am comparing 3 means (as b has 3 levels) at each combination of a x c (8 combinations). Here would I be dividing the alpha value by 3 or by the combinations x the different means hence 24).
Similarly, for axb at each level of c  am comparing 4 means (5 comparisons total) at each combination of a x b (6 total).Here would I be dividing the alpha value by 5 or by the combinations x the different means hence 30.
Alternatively would i be dividing the alpha value by the sum of all the above combinations?
Thanks
 A: I'll be the bunny who points out that not every set of scientific questions requires a 'correction' for multiplicity of tests and not every approach to statistical inference involves keeping track of type I errors. (Prepares self for down-votes!)
You wish to be sure that you are "not violating any key statistics theory", but if you make adjustments to p-values (or critical thresholds) then you can be assured that you will be violating the likelihood principle in order to comply with the repeated sampling principle. If you wish to behave as a pure frequentist then comply with the repeated sampling principle at all costs: adjust away. However, if you wish to deal directly with the evidence in your data then you cannot be a pure frequentist because you have to comply with the likelihood principle.
If you are interested in the evidence then it is helpful to know that the evidence itself is unaffected by multiplicity of testing (and by stopping rules). Of course if you wish to make a decision on the basis of the evidence it is perfectly OK to take multiplicity into account.
For your particular application I would imagine that you might be able to answer your substantive question using a hierarchical model instead of a bunch of frequentist error rate-adjusted hypothesis tests. (Most Bayesian methods comply with the likelihood principle.) Here's an example that might help you see the bigger picture: http://www.stat.columbia.edu/~gelman/research/published/multiple2f.pdf
A: Fitting one linear model (this is performed with ANOVA) can be viewed as omnibus test. So you are checking whether at least one of the relationships is significant. If ANOVA gives you p>0.05, then all post-hoc tests are redundant and are non-significant regardless of their p-values. When ANOVA p-value is significant, then you may want to perform post-hoc tests. One of the good ways to do this - Tukey HSD test. As for your design - to perform Bonferroni adjustment for multiple comparison you should divide alpha-value (for all results from this design) by the number of total comparisons - it is the number of levels in A + the number of levels in B + the number of levels in C. This sum is 2+3+4=9 as I've understand from your description. Note, that in example "AxB at each level of C" you don't compare 2 means 12 times, but compare 12 means two times, so number of comparisons is 2. But since further you'll test next issues (AxC at each level of B and so on), you should not divide one alpha in one issue by two and in another by three and by four, but all divide by nine.
A: As far as I know the p-value isn't divided by a correction factor, it's the Type I Error threshold value (alpha) which is adjusted by the number of comparisons being made. 
