0
$\begingroup$

Suppose some researcher does a hypothesis test at a 10% significance level, and the conclusion is to reject $ H_0 $. Then the researcher does the exact same test again, on the exact same dataset, but just lowers the significance level to 5%, and keeps lowering it until the result changes. If it still returns to reject $ H_0 $, are there any issues whatsoever with saying "there is a (e.g.) ~1.5% probability of type I error", as if the first test at 10% was never conducted?

Has the power or otherwise reliability of the test been affected by lowering the significance value on the same dataset?

$\endgroup$
2
  • 1
    $\begingroup$ The lowest-possible significance level at which rejection can occur is the p-value (up to a factor of $2$ for two-tailed tests). And yes, what you propose is a problem, though the "why" warrants an answer instead of a comment. $\endgroup$
    – Dave
    Jul 13, 2020 at 21:08
  • $\begingroup$ There's an issue with your characterization of a p-value as a "probability of type I error," because that is not what it is. At best the p-value is indirectly related to a hypothetical probability of a Type I error supposing the null hypothesis holds (along with many other attendant assumptions). Indeed, if--based on a small p-value--you decide to reject the null, you are flatly contradicting yourself by quoting a probability calculated on the assumption the null is true! $\endgroup$
    – whuber
    Jul 13, 2020 at 21:37

1 Answer 1

0
$\begingroup$

This is a form of the Texas Sharpshooter Fallacy.

If you just report a p-value and let the reader decide if it is significant or not (or even if they want to declare significance or not) and don't make any objective decisions/declarations based on the p-value, then that is reasonable.

But when you indicate that the test was done at a specific significance level, say 5%, then you are saying that before collecting the data (before randomization) there was a 5% chance of declaring significance/rejecting the null hypothesis if in fact the null hypothesis is True. In your scenario, with a true null hypothesis a p-value of 0.09 would have lead to a declaration of significance, so the probability of rejecting when the null is true is 10%, to say that it was 1.5% is dishonest.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.