# How to fix the threshold for statistical validity of p-values produced by ANOVAs?

I have run experiments on a group of users under two conditions, measuring the time it took users to finish their experiments. I used a cross-over design where half of the users started in the first conditions to end with the second, and the other half of the users did the other way around.

I analyze the data provided in a few different ANOVAs and find different p-values for my hypotheses. Some are below 0.05, some are below 0.01, some are over 0.05.

Do I need to fix an alpha level of statistical significance to be used in all my analysis, or can I report something like 'Hypothesis A is proven true at alpha level 0.05, while Hypothesis B is true at alpha level 0.01 (thus, possibly a stronger proof)'?

I don't know if I am being clear enough here. Let me know and I'll add details if needed.

Thanks.

• The statement "hypothesis A is proven true..." is not something statistics can be much help with in absolute terms. Statistics is much more useful at comparing different hypothesis, it would be better put in terms of "the data supports hypothesis B compared to hypothesis A" and other comparative statements. – probabilityislogic Jan 22 '11 at 18:13

Hey, but it seems you already looked at the results!

Usually, the risk of falsely rejecting the null (Type I error, or $\alpha$) should be decided before starting the analysis. Power might also be fixed to a given value (e.g., 0.80). At least, this is the "Neyman-Pearson" approach. For example, you might consider a risk of 5% ($\alpha=0.05$) for all your hypotheses, and if the tests are not independent you should consider correcting for multiple comparisons, using any single-step or step-down methods you like.

When reporting your results, you should indicate the Type I (and II, if applicable) error you considered (before seeing the results!), corrected or not for multiple comparisons, and give your p-values as p<.001 or p=.0047 for example.

Finally, I would say that your tests allow you to reject a given null hypothesis not to prove Hypothesis A or B. Moreover, what you describe as 0.001 being a somewhat stronger indication of an interesting deviation from the null than 0.05 is more in light with the Fisher approach to statistical hypothesis testing.

My advice would be to tread carefully with p-values if you didn't have a specific hypothesis in mind before you started the experiment. Adjusting p-values for multiple and "vaguely specified" hypothesis (e.g. not specifying the alternative hypothesis) is difficult.

I suppose the "purist" would tell you that this should be fixed prior to looking at the data (one of my lecturers call not doing this intellectual dishonesty), but I would only say this is appropriate for "confirmatory analysis" where a well defined model (or set of models) has been set prior to the data being seen.

If the analysis is more "exploratory" then I would not worry about precise level so much, rather try to find relationships and try to explain why they may be there (i.e. use the analysis to build a model). tentative hypothesis testing may be useful as an initial guide, but you would need to get more data to confirm your hypothesis.

A useful way to "get more data" without running another experiment is to "lock up" some portion of your data and use the rest to "explore" and then once you are confident of a potentially useful model, "test" your theory with the data you "locked up". NOTE: you can on do the "test" once!

• (+1) Your 2nd paragraph is particularly well formulated (I initially interpreted the question as an experimental study with a priori hypotheses); I also like the 3rd paragraph, and I recommend this approach when finding interesting but unexpected interaction effects. Finally, I don't know about any application of cross-validation in ANOVA designs: do you have any references? – chl Jan 23 '11 at 21:26
• @chl - cross-validation is different to what I meant by "lock up the data". this part would not be tested,looked at,analysed,etc.,etc. at all until your analysis was finished. Cross validation would be done with the remaining data. A good book on this (and many other things) is called The elements of Statistical Learning, but I don't have the full reference right now, but I can get it in a few weeks (when I go back to work). You could try "googling it". – probabilityislogic Jan 24 '11 at 12:00
• @chl - When you do cross validation, it is usually done with respect to prediction, and you need to specify a criterion for "good" and "bad" predictions, similar to a loss function in a decision theoretical framework. You also need to specify the competing models (I don't think cross-validation can do this for you). Cross validation then gives one way of deciding which model out of the alternatives you gave it was the best according to the criterion of "best" that you gave it. In ANOVA one usually isn't interested in prediction, but inference, so it may not be so useful in this case. – probabilityislogic Jan 24 '11 at 12:09
• ...continuing... although, having said that, the model which gives the "best" predictions does seem like a good model to investigate further (as oppose to the one which gives the "worst" predictions). – probabilityislogic Jan 24 '11 at 12:11
• Ah, but I know what CV is (btw, the ESL book is available here, www-stat.stanford.edu/~tibs/ElemStatLearn). I was just wondering whether you know references where CV is used in applied work. – chl Jan 24 '11 at 12:24