# Are there any disadvantages of performing a hypothesis test at the lowest possible significance level that still rejects the null hypothesis?

Suppose some researcher does a hypothesis test at a 10% significance level, and the conclusion is to reject $$H_0$$. Then the researcher does the exact same test again, on the exact same dataset, but just lowers the significance level to 5%, and keeps lowering it until the result changes. If it still returns to reject $$H_0$$, are there any issues whatsoever with saying "there is a (e.g.) ~1.5% probability of type I error", as if the first test at 10% was never conducted?

Has the power or otherwise reliability of the test been affected by lowering the significance value on the same dataset?

• The lowest-possible significance level at which rejection can occur is the p-value (up to a factor of $2$ for two-tailed tests). And yes, what you propose is a problem, though the "why" warrants an answer instead of a comment.
– Dave
Jul 13, 2020 at 21:08
• There's an issue with your characterization of a p-value as a "probability of type I error," because that is not what it is. At best the p-value is indirectly related to a hypothetical probability of a Type I error supposing the null hypothesis holds (along with many other attendant assumptions). Indeed, if--based on a small p-value--you decide to reject the null, you are flatly contradicting yourself by quoting a probability calculated on the assumption the null is true!
– whuber
Jul 13, 2020 at 21:37