I have run experiments on a group of users under two conditions, measuring the time it took users to finish their experiments. I used a cross-over design where half of the users started in the first conditions to end with the second, and the other half of the users did the other way around.
I analyze the data provided in a few different ANOVAs and find different p-values for my hypotheses. Some are below 0.05, some are below 0.01, some are over 0.05.
Do I need to fix an alpha level of statistical significance to be used in all my analysis, or can I report something like 'Hypothesis A is proven true at alpha level 0.05, while Hypothesis B is true at alpha level 0.01 (thus, possibly a stronger proof)'?
I don't know if I am being clear enough here. Let me know and I'll add details if needed.