1
$\begingroup$

I am new to statistical significance testing and i am wondering how the confidence level is set.

If for example I am conducting a test to prove that my method is better than an existing one, and i fail to reject the null hypothesis with a confidence interval of 95%, is it allowed to go back and lower the confidence interval to then reject the null hypothesis and verify that my method is better than the existing.

Any help is greatly appreciated

$\endgroup$
  • $\begingroup$ I don't see a legitimate way to change the significance level to get a result that you prefer. The hypothesis test has a significance level. It is the corresponding confidence interval that has a confidence level. That said, if you do a one-sided test rather than a two-sided test and would be satisfied with say a 0.1 significance level corresponding to a 90% confidence interval, that could possibly change the result. But these things need to be done before looking at the data and the results. $\endgroup$ – Michael Chernick May 9 '17 at 22:18
  • $\begingroup$ Thank you for your answer Micheal. If possible could you elaborate why we should set the confidence interval before looking at the result. Why can't we just run the experiment and lower the interval to the point where i can confirm my hypothesis. That is for example i've tested and could not verify my hypothesis with a 95% confidence interval, so i'll change it to 90 % and now i can verify my hypothesis $\endgroup$ – user89423 May 9 '17 at 22:58
1
$\begingroup$

The short answer is no. A big no-no is going back and re-adjusting your confidence intervals post-hoc (i.e. after you have done the experiment). See for example:http://blog.minitab.com/blog/michelle-paret/alphas-p-values-confidence-intervals-oh-my This is especially true when you are already using a lax threshold (95%), as in your case. In a case like yours, if your method is close to rejecting the null (e.g. .06) and the other method is downright terrible (e.g. .73) and if you know that rejecting the null hypothesis is the correct answer (e.g. from separate expert knowledge), I would look to see if the magnitude of the result in your method is at least pointing in the right direction, and comment on that.You could perhaps suggest that your method is on the right track, while the other method is not at all.

$\endgroup$
  • $\begingroup$ Thank you @user3923510, it cleared things up a bit for me. If possible could you elaborate as to why we can't go back and perform the same experiment with a different confidence interval to achieve the highest possible confidence interval? That is for example i've tested and could not verify my hypothesis with a 95% confidence interval, so i'll change it to 90 % and now i can verify my hypothesis. $\endgroup$ – user89423 May 9 '17 at 22:42
  • $\begingroup$ When you change the significance level after the fact, you are tailoring the statistical test to get the results you want, which is why it is important to decide on a statistical test (and confidence interval) before performing it. $\endgroup$ – Josh May 10 '17 at 1:08
0
$\begingroup$

I like user3923510's answer, there's just something I wanted to add based off your comments.

The goal of statistics is not to verify your hypothesis. The goal of statistics is to verify what the data is saying. If we think of simple classical statistics, we might perform an experiment, visualise the data, and then run an analysis to verify whether patterns observed in the visualisation are significantly different or not.

Positive results are not the only worthwhile results. Negative results are also telling you something about your data. Since you haven't provided details about your data, let me come up with something. Say there were two rivers. I hypothesise that the average weight of trout is going to be higher in river 1 because I know that river 1 is generally warmer than river 2. So I go and collect fish from each river and weigh them. I then perform a t-test to see whether average weight is significantly different, and I get a p-value of 0.085. Therefore the average weights of fish between the two rivers is the same.

This is my result though!! I do not now try to tweak the data to prove my original hypothesis. The statistics have done their job, and I have learn't something in the process. The weights do not differ! This is an interesting result and I will now design follow up experiments to investigate further.

Trying to force significant results is dangerous. You have your results. Trust them, use them, figure out what they mean.

$\endgroup$
  • $\begingroup$ Hi Conor, thank you for your reply. I agree that altering the data to achieve a given result is bad, i am not in the context of changing the data only the confidence interval. For your given example, if i perform a statistical significance test to verify with 95% confidence (interval) that the trouts are bigger in river 1, however i am not able to show this, so i try the experiment again, with a lower confidence interval. and find that with 90 % confidence (interval) i can determine that the trouts in river 1 are bigger than the trouts in river 2. I don't see anything wrong with this approach? $\endgroup$ – user89423 May 10 '17 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.