How to fix the threshold for statistical validity of p-values produced by ANOVAs? I have run experiments on a group of users under two conditions, measuring the time it took users to finish their experiments. I used a cross-over design where half of the users started in the first conditions to end with the second, and the other half of the users did the other way around. 
I analyze the data provided in a few different ANOVAs and find different p-values for my hypotheses. Some are below 0.05, some are below 0.01, some are over 0.05.
Do I need to fix an alpha level of statistical significance to be used in all my analysis, or can I report something like 'Hypothesis A is proven true at alpha level 0.05, while Hypothesis B is true at alpha level 0.01 (thus, possibly a stronger proof)'?
I don't know if I am being clear enough here. Let me know and I'll add details if needed.
Thanks.
 A: Hey, but it seems you already looked at the results!
Usually, the risk of falsely rejecting the null (Type I error, or $\alpha$) should be decided before starting the analysis. Power might also be fixed to a given value (e.g., 0.80). At least, this is the "Neyman-Pearson" approach. For example, you might consider a risk of 5% ($\alpha=0.05$) for all your hypotheses, and if the tests are not independent you should consider correcting for multiple comparisons, using any single-step or step-down methods you like. 
When reporting your results, you should indicate the Type I (and II, if applicable) error you considered (before seeing the results!), corrected or not for multiple comparisons, and give your p-values as p<.001 or p=.0047 for example. 
Finally, I would say that your tests allow you to reject a given null hypothesis not to prove Hypothesis A or B. Moreover, what you describe as 0.001 being a somewhat stronger indication of an interesting deviation from the null than 0.05 is more in light with the Fisher approach to statistical hypothesis testing.
A: My advice would be to tread carefully with p-values if you didn't have a specific hypothesis in mind before you started the experiment.  Adjusting p-values for multiple and "vaguely specified" hypothesis (e.g. not specifying the alternative hypothesis) is difficult.
I suppose the "purist" would tell you that this should be fixed prior to looking at the data (one of my lecturers call not doing this intellectual dishonesty), but I would only say this is appropriate for "confirmatory analysis" where a well defined model (or set of models) has been set prior to the data being seen.
If the analysis is more "exploratory" then I would not worry about precise level so much, rather try to find relationships and try to explain why they may be there (i.e. use the analysis to build a model).  tentative hypothesis testing may be useful as an initial guide, but you would need to get more data to confirm your hypothesis.
A useful way to "get more data" without running another experiment is to "lock up" some portion of your data and use the rest to "explore" and then once you are confident of a potentially useful model, "test" your theory with the data you "locked up".  NOTE: you can on do the "test" once!
