Proper frequentist procedure says that we should establish a sampling plan, statistical analysis and significance level before we start an experiment. We then report whether or not we reject H0 for the chosen level of significance.

What if we do the same and don't choose the significance level beforehand. We simply look for the lowest significance level we can get away with, and report that, stating that we can reject the null hypothesis with the given level as significance. The lower this value is, the more weight we can give to our analysis.

If we are upfront about following this procedure, is this honest? If not, where is the problem? I'm having trouble figuring out what the issue is, and why people don't report like this in general.

  • $\begingroup$ I don't think this is a question of dishonesty, but the completeness. The analysis isn't complete unless you say whether your results were significant. A reader (most of whom aren't statisticians) ultimately wants to know the significance of results. If we don't set a universally agreed-upon boundary, how is the reader supposed to interpret the results? Sure, a p-value of 6% is not as bad a p-value of 70%, but this relative difference is meaningless when the reader is just interested in a yes/no answer. $\endgroup$ – rocinante Jun 27 '14 at 17:17
  • 4
    $\begingroup$ You haven't done a significance test (no rejection rule, no decision, no conclusion), but simply explained to the reader how they might do one as soon as they choose a significance level. $\endgroup$ – Glen_b -Reinstate Monica Jun 27 '14 at 17:59
  • $\begingroup$ @Glen_b So that means that there's no problem with choosing your significance level post-hoc? $\endgroup$ – Peter Jun 27 '14 at 19:03
  • $\begingroup$ I don't see how one would conclude that from what I said. What is critical is that the significance level is not selected with any reference to the data (including with any reference to the p-value, since that's a function of the data). To be explicit: there is a problem with doing that post hoc. $\endgroup$ – Glen_b -Reinstate Monica Jun 27 '14 at 23:54

It really depends on what you trying to accomplish from your analysis. For instance, if you're a medical researcher trying to test out the effects of a new drug, you'll need to be able to establish that it actually works before you can bring it to market. So you'll want to pick a low significance level and actually hold yourself to it, because other people are going to hold you to it as well.

If you're not that concerned with if something would pass or fail a significance test, then it's fine to just run the analysis and report what the p-value is. As noted in comments, then you're leaving it up to the reader to decide if they are convinced or not. I think it is much better practice in these cases to report the actual p-value: knowing something is 0.01 or 0.045 is a lot more helpful for the reader than knowing that it's just <0.05.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.