Are there any disadvantages of performing a hypothesis test at the lowest possible significance level that still rejects the null hypothesis?

Question

Suppose some researcher does a hypothesis test at a 10% significance level, and the conclusion is to reject $ H_0 $. Then the researcher does the exact same test again, on the exact same dataset, but just lowers the significance level to 5%, and keeps lowering it until the result changes. If it still returns to reject $ H_0 $, are there any issues whatsoever with saying "there is a (e.g.) ~1.5% probability of type I error", as if the first test at 10% was never conducted?

Has the power or otherwise reliability of the test been affected by lowering the significance value on the same dataset?

Answer 1

This is a form of the Texas Sharpshooter Fallacy.

If you just report a p-value and let the reader decide if it is significant or not (or even if they want to declare significance or not) and don't make any objective decisions/declarations based on the p-value, then that is reasonable.

But when you indicate that the test was done at a specific significance level, say 5%, then you are saying that before collecting the data (before randomization) there was a 5% chance of declaring significance/rejecting the null hypothesis if in fact the null hypothesis is True. In your scenario, with a true null hypothesis a p-value of 0.09 would have lead to a declaration of significance, so the probability of rejecting when the null is true is 10%, to say that it was 1.5% is dishonest.