The meaning of two sided (double tailed) p-values

A p-value is "the probability for a given statistical model that, when the null hypothesis is true, the statistical summary would be the same as or of greater magnitude than the actual observed results." Source

Thus, the p-value is defined as Pr(X≥x|H) for a one sided (right tailed) test. For a two sided (double tailed) test, this probability is doubled, meaning p = 2 * Pr(X≥x|H) and then compared to alpha (e.g .05). However, how can this two sided p-value be interpreted as the probability as defined above? The real probability of getting a stat. summary of greater magnitude than the observed result (e.g. +2) is Pr(X≥2|H) and not 2 * Pr(X≥2|H). Yet, statisticians double the right-hand p-value. There is, of course, an error probability on the left side too (alpha/2), but this should be of no concern for us, as our observed value is positive and thus creates a probability only on its right side to +infinity below the curve.

The question, therefore, is: Would it not be more mathematically accurate to compare Pr(X≥2|H) to alpha/2 (=.025) instead of 2 * Pr(X≥2|H) to alpha (=.05) in a two sided test? The results would of course be the same in symmetric distributions, but not in asymmetric ones!

Furthermore, if we say that p for double tailed tests is indeed 2 * Pr(X≥x|H), then this p is not the probability of X being of greater magnitude than x, but rather this probability times 2.

• One of my intentions in writing stats.stackexchange.com/a/130772/919 was to help people understand two-tailed tests (and even more complicated situations), so please take a look. Your quotation, although from a reliable source, is correct only in certain textbook situations. It is not correct as a characterization of p-values generally. – whuber Jun 29 '18 at 20:18

I would prefer to consider extreme rather than magnitude

If you are expecting the value $0$ for something thought to have a symmetric distribution but in fact see the value $2$

then other values that you might have seen which would be as extreme as $2$ or more extreme than $2$ would include all values greater than or equal to $2$ and all values less than or equal to $-2$

so the probability of seeing $2$ or a more extreme value can be expressed as $\mathbb P(|X| \ge2) = \mathbb P(X \ge2) + \mathbb P(X \le -2) = 2\mathbb P(X \ge2)$ by symmetry

In asymmetric distributions, you need to decide what extreme means in the context of your experiment

If you had a lightbulb which had an exponentially distributed lifetime L expected to be $1000$ hours, you might decide that extreme means that seeing a lifetime of $4000$ hours or more would as extreme as seeing a lifetime of about $18.5$ hours or less, or you might take a different view. You could still say $\mathbb P(L \ge 4000) + \mathbb P(L \le 18.5) \approx 2\mathbb P(L \ge 4000) \approx 0.0366$

You should decide what you mean by as extreme or more extreme before you see the actual observation, and this is especially important with a distribution believed to be asymmetric