A p-value is "the probability for a given statistical model that, when the null hypothesis is true, the statistical summary would be the same as or of greater magnitude than the actual observed results." Source

Thus, the p-value is defined as Pr(X≥x|H) for a one sided (right tailed) test. For a two sided (double tailed) test, this probability is doubled, meaning p = 2 * Pr(X≥x|H) and then compared to alpha (e.g .05). However, how can this two sided p-value be interpreted as the probability as defined above? The real probability of getting a stat. summary of greater magnitude than the observed result (e.g. +2) is Pr(X≥2|H) and not 2 * Pr(X≥2|H). Yet, statisticians double the right-hand p-value. There is, of course, an error probability on the left side too (alpha/2), but this should be of no concern for us, as our observed value is positive and thus creates a probability only on its right side to +infinity below the curve.

The question, therefore, is: Would it not be more mathematically accurate to compare Pr(X≥2|H) to alpha/2 (=.025) instead of 2 * Pr(X≥2|H) to alpha (=.05) in a two sided test? The results would of course be the same in symmetric distributions, but not in asymmetric ones!

Furthermore, if we say that p for double tailed tests is indeed 2 * Pr(X≥x|H), then this p is not the probability of X being of greater magnitude than x, but rather this probability times 2.

  • 2
    $\begingroup$ One of my intentions in writing stats.stackexchange.com/a/130772/919 was to help people understand two-tailed tests (and even more complicated situations), so please take a look. Your quotation, although from a reliable source, is correct only in certain textbook situations. It is not correct as a characterization of p-values generally. $\endgroup$ – whuber Jun 29 '18 at 20:18

I would prefer to consider extreme rather than magnitude

If you are expecting the value $0$ for something thought to have a symmetric distribution but in fact see the value $2$

then other values that you might have seen which would be as extreme as $2$ or more extreme than $2$ would include all values greater than or equal to $2$ and all values less than or equal to $-2$

so the probability of seeing $2$ or a more extreme value can be expressed as $\mathbb P(|X| \ge2) = \mathbb P(X \ge2) + \mathbb P(X \le -2) = 2\mathbb P(X \ge2)$ by symmetry

In asymmetric distributions, you need to decide what extreme means in the context of your experiment

If you had a lightbulb which had an exponentially distributed lifetime L expected to be $1000$ hours, you might decide that extreme means that seeing a lifetime of $4000$ hours or more would as extreme as seeing a lifetime of about $18.5$ hours or less, or you might take a different view. You could still say $\mathbb P(L \ge 4000) + \mathbb P(L \le 18.5) \approx 2\mathbb P(L \ge 4000) \approx 0.0366$

You should decide what you mean by as extreme or more extreme before you see the actual observation, and this is especially important with a distribution believed to be asymmetric


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.