Calculating 'k' in the bonferroni procedure when there are both post-hoc and planned constrasts I have run an experiment with a 2 x 2 factorial design. From what I understand there is no need to adjust the alpha-level for multiple comparisons because all main-effects and interactions are planned.
However I have had a reviewer come back and suggest analyses of some simple effects and as a result I am starting to wonder about error-correction for multiple post-hoc comparisons. 
My question is when calculating something like a Bonferroni error correction, how many comparisons should be in the formula? Should I only correct for the post-hoc tests or does doing post-hoc tests mean I should include all planned contrasts as well? So, if performing the bonferroni procedure, what should the k in $\alpha$/$k$ be for a 2 x 2 ANOVA with 2 tests of simple effects?
 A: It's not so straightforward to answer because developing formal rules for the number of corrections would require to invoke Bayesian approaches.
Basically, the idea is not that you correct on whether a test was "planned" or not. Corrections should be done for all tests, if you want to maintain the type I error across the experiment. However, if you add known predictors to your model, you are pretty sure that the null hypothesis is false for that predictor, so you do not expect that test to contribute to the overall type I error. Blind post-hoc tests, on the other hand, are not supported by external evidence, and you would be more strict towards them. 
To give an applied example, in large-throughput biology, an experiment can be designed to test 20,000 genes - all of these tests are planned, yet you apply a correction because the probability of null being true is high for each gene. In an epidemiological analysis of lung cancer one would include smoking without worrying about correcting for that predictor.  
In short, I would suggest that you correct for the number of post-hoc tests only, and discuss any further issues in the text of the paper.
