# Adjusting the confidence level after the experiment

I am new to statistical significance testing and i am wondering how the confidence level is set.

If for example I am conducting a test to prove that my method is better than an existing one, and i fail to reject the null hypothesis with a confidence interval of 95%, is it allowed to go back and lower the confidence interval to then reject the null hypothesis and verify that my method is better than the existing.

Any help is greatly appreciated

• I don't see a legitimate way to change the significance level to get a result that you prefer. The hypothesis test has a significance level. It is the corresponding confidence interval that has a confidence level. That said, if you do a one-sided test rather than a two-sided test and would be satisfied with say a 0.1 significance level corresponding to a 90% confidence interval, that could possibly change the result. But these things need to be done before looking at the data and the results. – Michael R. Chernick May 9 '17 at 22:18
• Thank you for your answer Micheal. If possible could you elaborate why we should set the confidence interval before looking at the result. Why can't we just run the experiment and lower the interval to the point where i can confirm my hypothesis. That is for example i've tested and could not verify my hypothesis with a 95% confidence interval, so i'll change it to 90 % and now i can verify my hypothesis – user89423 May 9 '17 at 22:58

The short answer is no. A big no-no is going back and re-adjusting your confidence intervals post-hoc (i.e. after you have done the experiment). See for example:http://blog.minitab.com/blog/michelle-paret/alphas-p-values-confidence-intervals-oh-my This is especially true when you are already using a lax threshold (95%), as in your case. In a case like yours, if your method is close to rejecting the null (e.g. .06) and the other method is downright terrible (e.g. .73) and if you know that rejecting the null hypothesis is the correct answer (e.g. from separate expert knowledge), I would look to see if the magnitude of the result in your method is at least pointing in the right direction, and comment on that.You could perhaps suggest that your method is on the right track, while the other method is not at all.

• Thank you @user3923510, it cleared things up a bit for me. If possible could you elaborate as to why we can't go back and perform the same experiment with a different confidence interval to achieve the highest possible confidence interval? That is for example i've tested and could not verify my hypothesis with a 95% confidence interval, so i'll change it to 90 % and now i can verify my hypothesis. – user89423 May 9 '17 at 22:42
• When you change the significance level after the fact, you are tailoring the statistical test to get the results you want, which is why it is important to decide on a statistical test (and confidence interval) before performing it. – Josh May 10 '17 at 1:08