6
$\begingroup$

Suppose you are using $\alpha = 0.01$ as the significance level. IF you are not getting significant results, is it better to increase the significance level to $\alpha = 0.05$?

$\endgroup$
  • 15
    $\begingroup$ Absolutely. And if your results aren't significant at 0.05, you can always go to 0.10 and even 0.20: whatever it takes to get the answer you need, right? :-) $\endgroup$ – whuber Mar 6 '12 at 20:16
  • 7
    $\begingroup$ A farmer had lots of targets painted on the side of his barn with bullet holes perfectly centered in the bullseye. As he explained, "I shoot first and paint the target afterwards." $\endgroup$ – Dilip Sarwate Mar 6 '12 at 20:33
  • 2
    $\begingroup$ The Earth Is Round (p < .05) (or spherical) :) $\endgroup$ – chl Mar 6 '12 at 23:36
  • $\begingroup$ Like it or not, this practice is widespread and even if you know its wrong, you're often pressured to do it. People want validation of their hunches and will tailor methods to do so; asides from refusing (and getting replaced) how do you stand up to that? $\endgroup$ – user4673 Mar 7 '12 at 18:31
  • $\begingroup$ See if you can find a good reference for the culture of "Null Hypothesis Significance Testing" - There isn't many. In fact you will find that there are many more articles which criticise this practise. I'm pretty sure its a blend of Neyman-Pearson hypothesis testing with Fishers p-values. I'm also pretty sure niether said their method should be used this way. $\endgroup$ – probabilityislogic Oct 7 '12 at 17:14
22
$\begingroup$

Expanding on @EpiGrad 's answer (which is a good answer):

  1. There are many reasons to ignore p-values altogether: Principally, they answer a question we are very rarely interested in.

  2. If you are going to use p-values, using them as cutoffs often makes little sense.

  3. If you are going to use them as cut-offs, you should decide on the cutoff before the analysis

  4. Making a more stringent cutoff for type I error means lower power (more type II error). Typical values are .05 for type I and .20 for type II (power = .8). But there is no reason why type II errors are necessarily less bad than type I errors. Suppose you develop a drug that treats a disease that is terminal and rapidly so (e.g. something like Ebola). You test it.

    Type I error - you say the drug does something when it doesn't, and then give a useless drug to dying people.

    Type II error - you say the drug does nothing when it does something, and you fail to give a beneficial drug to dying people.

    Which is worse? Type II, by my book.

To quote Prof. David Cox

There are no routine statistical questions, only questionable statistical routines

$\endgroup$
  • 4
    $\begingroup$ Except, the type I scenario could also end up in significant patient harm if the patient doesn't have a terminal condition and the (falsely positive) treatment has significant side effects. Both type I and type II are bad, depending on the context. $\endgroup$ – pmgjones Mar 6 '12 at 20:43
  • 2
    $\begingroup$ @pmgjones: Yes, I believe that's the point that Peter Flom is making. He wrote that "there is no reason why type II errors are necessarily less bad than type I errors" (emphasis mine), then gave one example situation where he would consider type II errors to be worse. $\endgroup$ – ruakh Mar 6 '12 at 22:06
  • 1
    $\begingroup$ True - I had missed that part of his message. Sorry Peter! $\endgroup$ – pmgjones Mar 6 '12 at 22:17
  • 1
    $\begingroup$ +1 for many reasons to ignore them completely. If I could give it another +1, I would for the emphasis on Type II errors still being errors. $\endgroup$ – Fomite Mar 7 '12 at 0:14
  • 1
    $\begingroup$ This is reminiscence of the trade off between Precision and Recall. $\endgroup$ – Itamar Mar 12 '12 at 19:16
17
$\begingroup$

Just if you're not getting significant results? No. Fiddling with the significance level after the experiment is conducted and the results are known is never good practice.

There are circumstances where you might chose a more relaxed p-value, but doing it post hoc is just a bad idea.

$\endgroup$
6
$\begingroup$

On the one hand it is somewhat artificial to discount a variable because its p value is higher than 0.01 (that's an unusually stringent criterion). How you get there may be more important than what is the ultimate significance level. A variable that is well grounded on logic or causal links with an acceptable p value may be much more meaningful than a variable with a lower p value but with no meaningful logic supporting it.

If you are dealing with hypothesis testing, watch out that the statistical significance is in good part just a function of your sample size. A large sample size will translate into a low standard error and higher statistical significance. And, this process is somewhat artificial as large samples will render immaterial differences statistically significant. If you are dealing within such a domain I recommend you move towards an Effect Size method where the unit of statistical distance is not Standard Error but instead Standard Deviation. And, the latter can't be manipulated by sample size.

$\endgroup$
  • $\begingroup$ 0.01 is only small by conventional standards. If you use BIC to assess significance, the equivalent cutoff can be much smaller. For example a sample size of 10,000 means using a p-value of something like 0.001. $\endgroup$ – probabilityislogic Oct 7 '12 at 17:06
-1
$\begingroup$

I found myself asking the same question and came here looking for good arguments, and I have to say I'm not convinced. Let me explain why I think it is fine to change significance level after an experiment.

  1. Whatever you calculate using your chosen significance level a, will have the same value whether you choose it before or after the experiment. In other words, given some results, it is not possible to infer if the choice of a was done before or after.

  2. Ethics only becomes an issue if you are unclear about what you are doing. If you say that you measured a significant effect without specifying what you mean by that, it can be misleading. This is irrelevant, though, if you state, as you should, what a you chose. If your data is available, all the better, since your calculations can be easily reproduced by anyone interested.

  3. If someone decides to use your results, they will have their own criteria to what magnitude of type I and type II errors they are willing to accept. If you get a significant result using a higher a, it can still be an useful information. It is up to whoever is using your information to decide if the chosen a is low enough for them or not.

To illustrate the last argument, I'll draw from the anecdote from @dilip-sarwate. Say a farmer did paint the targets on the side of his barn before and tried to shoot at them. He sees then that the bullets landed outside the targets, but only by a bit. If he decides, afterwards, to increase the radii of the targets, one could call it cheating, but I think it is still valid information: that is, he is not as accurate as he expected to be, but he is still somewhat accurate and one could even say by how much. You then replace an useless result, 'the farmer is not good with a gun', with a more concrete one 'the farmer is accurate to within 5 cm'.

$\endgroup$
  • $\begingroup$ Whoever downvoted, could you elaborate on your reasoning? $\endgroup$ – mper Sep 19 '18 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.