Is it permissible to run post-hoc tests after a non-significant ANOVA?

Question

We ran a 3-way ANOVA getting a non-significant result, but in doing so we saw that one condition looked like it had a strong effect. We tried a Bonferroni-corrected t-test of just that group, and it turned out significant. Is it valid to consider this? Is there some other correction needed to deal with the fact that we tested that group only after we saw that it "looked" like an effect?

(And does the answer to this sort of question depend on between vs. within factors? We have a mix. I ask this because SPSS doesn't seem to give the option for post hoc tests under the GLM procedure except for between-factor effects. Not the best way to learn, but it's a clue, nonetheless.)

Answer 1

It depends on what you plan to do with the result.

If you want to form a conclusion regarding a null hypothesis and have a specifiable rate of false positives (i.e. you want a Neyman-Pearson hypothesis test) then no, you can't do it. Spuriously 'significant' results turn up all the time and any result that looks significant will quite likely turn out to be statistically significant even when it is not real. Neyman-Pearson analysis allows you to test PREDEFINED hypotheses with PREDETERMINED analyses.

If you want to use your data to help form hypotheses to test with new experiments (i.e. use a Fisherian approach) then yes, go ahead and test! The fact that you cannot reliably test an hypothesis that is formed on the basis of seeing a dataset using that same dataset does not mean that the dataset cannot point to something worthy of a follow-up experiment.