6
$\begingroup$

How do you report results that give you a p-value of 0.08 for a two-tailed test and a p-value of 0.04 for a one-tailed test? Generalizing the issue, if you obtain a p-value between 0.05 and 0.9999 from a two-tailed test, does it make sense to proceed with a one-tailed test in the direction whose p-value is less than 0.05?

$\endgroup$
12
$\begingroup$

You don't choose a one-tailed test based on near-significance in a two-tailed test.

You don't choose the direction of a one-tailed test based on directional information from the data.

Or at the least, if you do those things, you must also double the resulting p-value.

A one tailed test - if you do one at all - must be based on prior considerations, in place before you know what is in the data. If this is not the case, the significance levels (and p-values) are meaningless.

$\endgroup$
  • 1
    $\begingroup$ Assume two researchers perform the same exact experiment but with different hypotheses. One uses a two-tailed test while the other uses a one-tailed test. They both then publish their results (similar to the original question). A third-party scientist then comes in to do his own research. How will the scientist interpret the two different results? $\endgroup$ – Rudolph Jul 15 '14 at 20:13
  • 4
    $\begingroup$ This is an entirely different question, but if there was no good reason (a priori) for the third party to use a one tailed test, they would have to double the one-tailed p-value. If there was a good reason to, they could halve the two-tailed one. It would probably make more sense to look at the effect sizes - e.g. look at the difference in means, and the standard error of the difference in means and compare the two sets of results in that way. $\endgroup$ – Glen_b Jul 15 '14 at 20:23
  • $\begingroup$ I have the same question as @Rudolph. In particular, if we always do two one-sided tests at the same time, do we still need double p-value? $\endgroup$ – ibread Feb 4 '18 at 5:48
  • $\begingroup$ I kind of understand now. If we run two one-sided tests at the same time using the same alpha as two-sided one, but pick one of the one-sided tests to report, essentially we'll make twice Type-I errors in the long run as it is the sum of Type-I errors in either direction. @Glen_b is this right? $\endgroup$ – ibread Feb 4 '18 at 6:00
  • 1
    $\begingroup$ Running two tests and then reporting the most significant one is the same as looking at the data to see which one-sided test to choose, and yes, that's effectively doubling the type I error rate for any test where the two one sided alternatives are mutually exclusive (it is not always the case that the two one-sided alternatives are mutually exclusive -- a Kolmogorov-Smirnov is one example of an exception) $\endgroup$ – Glen_b Feb 4 '18 at 6:29
8
$\begingroup$

Report the results that correspond to your hypothesis, which should be one- or two-tailed, not both. You should be able to decide which is appropriate on a theoretical basis before performing the test. Once you've decided, report the p value as you calculated it. If it's very small, consider the advice in responses to this question: How should tiny $p$-values be reported? (and why does R put a minimum on 2.22e-16?) If you are using the Neyman–Pearson approach to interpreting your p value, you probably know how to decide whether to reject or retain your null hypothesis based on the false positive error rate, which you must also choose in advance.

It is incorrect to apply a one-sided test following a two-sided test of the otherwise-equivalent null hypothesis. Again, either one or the other is appropriate depending on your theoretical aim, not both. If a two-sided test is appropriate, you're using the Neyman–Pearson framework, and you fail to reject the null, then that is your result. If that doesn't suit your purposes, you can replicate the study anyway and see how it turns out the next time, but don't fail to report your first null result even if the second rejects the null. That is one of the primary causes of the file drawer effect, a meta-analyst's worst nightmare.

For more on understanding the difference between one- and two-tailed tests, see:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.