Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite
1

Suppose you are using $\alpha = 0.01$ as the significance level. IF you are not getting significant results, is it better to increase the significance level to $\alpha = 0.05$?

share|improve this question
11  
Absolutely. And if your results aren't significant at 0.05, you can always go to 0.10 and even 0.20: whatever it takes to get the answer you need, right? :-) – whuber♦ Mar 6 '12 at 20:16
4  
A farmer had lots of targets painted on the side of his barn with bullet holes perfectly centered in the bullseye. As he explained, "I shoot first and paint the target afterwards." – Dilip Sarwate Mar 6 '12 at 20:33
1  
The Earth Is Round (p < .05) (or spherical) :) – chl♦ Mar 6 '12 at 23:36
Like it or not, this practice is widespread and even if you know its wrong, you're often pressured to do it. People want validation of their hunches and will tailor methods to do so; asides from refusing (and getting replaced) how do you stand up to that? – user4673 Mar 7 '12 at 18:31
See if you can find a good reference for the culture of "Null Hypothesis Significance Testing" - There isn't many. In fact you will find that there are many more articles which criticise this practise. I'm pretty sure its a blend of Neyman-Pearson hypothesis testing with Fishers p-values. I'm also pretty sure niether said their method should be used this way. – probabilityislogic Oct 7 '12 at 17:14

3 Answers

up vote 19 down vote

Expanding on @EpiGrad 's answer (which is a good answer):

  1. There are many reasons to ignore p-values altogether: Principally, they answer a question we are very rarely interested in.

  2. If you are going to use p-values, using them as cutoffs often makes little sense.

  3. If you are going to use them as cut-offs, you should decide on the cutoff before the analysis

  4. Making a more stringent cutoff for type I error means lower power (more type II error). Typical values are .05 for type I and .20 for type II (power = .8). But there is no reason why type II errors are necessarily less bad than type I errors. Suppose you develop a drug that treats a disease that is terminal and rapidly so (e.g. something like Ebola). You test it.

    Type I error - you say the drug does something when it doesn't, and then give a useless drug to dying people.

    Type II error - you say the drug does nothing when it does something, and you fail to give a beneficial drug to dying people.

    Which is worse? Type II, by my book.

To quote Prof. David Cox

There are no routine statistical questions, only questionable statistical routines

share|improve this answer
4  
Except, the type I scenario could also end up in significant patient harm if the patient doesn't have a terminal condition and the (falsely positive) treatment has significant side effects. Both type I and type II are bad, depending on the context. – pmgjones Mar 6 '12 at 20:43
1  
@pmgjones: Yes, I believe that's the point that Peter Flom is making. He wrote that "there is no reason why type II errors are necessarily less bad than type I errors" (emphasis mine), then gave one example situation where he would consider type II errors to be worse. – ruakh Mar 6 '12 at 22:06
1  
True - I had missed that part of his message. Sorry Peter! – pmgjones Mar 6 '12 at 22:17
1  
+1 for many reasons to ignore them completely. If I could give it another +1, I would for the emphasis on Type II errors still being errors. – EpiGrad Mar 7 '12 at 0:14
1  
This is reminiscence of the trade off between Precision and Recall. – Itamar Mar 12 '12 at 19:16
show 1 more comment
up vote 15 down vote

Just if you're not getting significant results? No. Fiddling with the significance level after the experiment is conducted and the results are known is never good practice.

There are circumstances where you might chose a more relaxed p-value, but doing it post hoc is just a bad idea.

share|improve this answer
1  
+1 for the restraint... – gung Mar 6 '12 at 20:14
up vote 5 down vote

On one hand it is somewhat artificial to discount a variable because its p value is higher than 0.01 (that's an unusually stringent criteria). How you get there may be more important than what is the ultimate significance level. A variable that is well grounded on logic or causal links with an acceptable p value may be much more meaningful than a variable with a lower p balue but with no meaningful logic supporting it.

If you are dealing with hypothesis testing watch out that the statistical significance is in good part just a function of your sample size. A large sample size will translate into a low standard error and higher statistical significance. And, this process is somewhat artificial as large samples will render immaterial differences statistically significant. If you are dealing within such a domain I recommend you move towards an Effect Size method where the unit of statistical distance is not Standard Error but instead Standard Deviation. And, the latter can't be manipulated by sample size.

share|improve this answer
0.01 is only small by conventional standards. If you use BIC to assess significance, the equivalent cutoff can be much smaller. For example a sample size of 10,000 means using a p-value of something like 0.001. – probabilityislogic Oct 7 '12 at 17:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.