I am comparing measurements on a test group relative to a control group in three different environmental conditions. I am interested in both differences between environmental conditions and differences between test and control groups. I ran a two-way ANOVA with an interaction term and looked at pairwise comparisons when terms were significant.
When the interaction term was significant the Tukey HSD function in R automatically outputs all comparisons. Comparisons between test and test groups on different environmental conditions, comparisons between test and control groups on different environmental conditions and so forth. Needless to say, this resulted in a large number of tests to correct for.
My adviser thinks that I should only do three tests to compare test group to control group on each environmental condition (and then only adjust for three tests). I think that because I am interested in differences between environmental conditions in this study, I should run most tests. If I wasn't interested in differences in environmental conditions it should be a nested ANOVA, right? You can see from graphs that the interaction term come from differences between test and control groups on two ecological sites, but it doesn't seem valid to just only run comparisons between groups you 'suspect' will be different. The ones I'm not sure I care about are differences test and control groups on two different environmental conditions.
Is it valid to only run comparisons between groups you are interested in to reduce the number of tests you have to adjust the Tukey HSD p-value for or should you run comparisons on all combinations of groups.
Thanks for help in advance.
