# Is it ever good to increase significance level?

Suppose you are using $\alpha = 0.01$ as the significance level. IF you are not getting significant results, is it better to increase the significance level to $\alpha = 0.05$?

-
Absolutely. And if your results aren't significant at 0.05, you can always go to 0.10 and even 0.20: whatever it takes to get the answer you need, right? :-) – whuber Mar 6 '12 at 20:16
A farmer had lots of targets painted on the side of his barn with bullet holes perfectly centered in the bullseye. As he explained, "I shoot first and paint the target afterwards." – Dilip Sarwate Mar 6 '12 at 20:33
– chl Mar 6 '12 at 23:36
Like it or not, this practice is widespread and even if you know its wrong, you're often pressured to do it. People want validation of their hunches and will tailor methods to do so; asides from refusing (and getting replaced) how do you stand up to that? – user4673 Mar 7 '12 at 18:31
See if you can find a good reference for the culture of "Null Hypothesis Significance Testing" - There isn't many. In fact you will find that there are many more articles which criticise this practise. I'm pretty sure its a blend of Neyman-Pearson hypothesis testing with Fishers p-values. I'm also pretty sure niether said their method should be used this way. – probabilityislogic Oct 7 '12 at 17:14

1. There are many reasons to ignore p-values altogether: Principally, they answer a question we are very rarely interested in.

2. If you are going to use p-values, using them as cutoffs often makes little sense.

3. If you are going to use them as cut-offs, you should decide on the cutoff before the analysis

4. Making a more stringent cutoff for type I error means lower power (more type II error). Typical values are .05 for type I and .20 for type II (power = .8). But there is no reason why type II errors are necessarily less bad than type I errors. Suppose you develop a drug that treats a disease that is terminal and rapidly so (e.g. something like Ebola). You test it.

Type I error - you say the drug does something when it doesn't, and then give a useless drug to dying people.

Type II error - you say the drug does nothing when it does something, and you fail to give a beneficial drug to dying people.

Which is worse? Type II, by my book.

To quote Prof. David Cox

There are no routine statistical questions, only questionable statistical routines

-
Except, the type I scenario could also end up in significant patient harm if the patient doesn't have a terminal condition and the (falsely positive) treatment has significant side effects. Both type I and type II are bad, depending on the context. – pmgjones Mar 6 '12 at 20:43
@pmgjones: Yes, I believe that's the point that Peter Flom is making. He wrote that "there is no reason why type II errors are necessarily less bad than type I errors" (emphasis mine), then gave one example situation where he would consider type II errors to be worse. – ruakh Mar 6 '12 at 22:06
True - I had missed that part of his message. Sorry Peter! – pmgjones Mar 6 '12 at 22:17
+1 for many reasons to ignore them completely. If I could give it another +1, I would for the emphasis on Type II errors still being errors. – Fomite Mar 7 '12 at 0:14
This is reminiscence of the trade off between Precision and Recall. – Itamar Mar 12 '12 at 19:16

Just if you're not getting significant results? No. Fiddling with the significance level after the experiment is conducted and the results are known is never good practice.

There are circumstances where you might chose a more relaxed p-value, but doing it post hoc is just a bad idea.

-
+1 for the restraint... – gung Mar 6 '12 at 20:14

On the one hand it is somewhat artificial to discount a variable because its p value is higher than 0.01 (that's an unusually stringent criterion). How you get there may be more important than what is the ultimate significance level. A variable that is well grounded on logic or causal links with an acceptable p value may be much more meaningful than a variable with a lower p value but with no meaningful logic supporting it.

If you are dealing with hypothesis testing, watch out that the statistical significance is in good part just a function of your sample size. A large sample size will translate into a low standard error and higher statistical significance. And, this process is somewhat artificial as large samples will render immaterial differences statistically significant. If you are dealing within such a domain I recommend you move towards an Effect Size method where the unit of statistical distance is not Standard Error but instead Standard Deviation. And, the latter can't be manipulated by sample size.

-
0.01 is only small by conventional standards. If you use BIC to assess significance, the equivalent cutoff can be much smaller. For example a sample size of 10,000 means using a p-value of something like 0.001. – probabilityislogic Oct 7 '12 at 17:06