# Establishing significance levels post-hoc

Proper frequentist procedure says that we should establish a sampling plan, statistical analysis and significance level before we start an experiment. We then report whether or not we reject H0 for the chosen level of significance.

What if we do the same and don't choose the significance level beforehand. We simply look for the lowest significance level we can get away with, and report that, stating that we can reject the null hypothesis with the given level as significance. The lower this value is, the more weight we can give to our analysis.

If we are upfront about following this procedure, is this honest? If not, where is the problem? I'm having trouble figuring out what the issue is, and why people don't report like this in general.

• I don't think this is a question of dishonesty, but the completeness. The analysis isn't complete unless you say whether your results were significant. A reader (most of whom aren't statisticians) ultimately wants to know the significance of results. If we don't set a universally agreed-upon boundary, how is the reader supposed to interpret the results? Sure, a p-value of 6% is not as bad a p-value of 70%, but this relative difference is meaningless when the reader is just interested in a yes/no answer. – rocinante Jun 27 '14 at 17:17
• You haven't done a significance test (no rejection rule, no decision, no conclusion), but simply explained to the reader how they might do one as soon as they choose a significance level. – Glen_b -Reinstate Monica Jun 27 '14 at 17:59
• @Glen_b So that means that there's no problem with choosing your significance level post-hoc? – Peter Jun 27 '14 at 19:03
• I don't see how one would conclude that from what I said. What is critical is that the significance level is not selected with any reference to the data (including with any reference to the p-value, since that's a function of the data). To be explicit: there is a problem with doing that post hoc. – Glen_b -Reinstate Monica Jun 27 '14 at 23:54