26
$\begingroup$

My situation is as follows: I want, through a Monte-Carlo study, to compare $p$-values of two different tests for statistical significance of an estimated parameter (null is "no effect - parameter is zero", and the implied alternative is "parameter is not zero"). Test A is the standard "independent two-sample t-test for equality of means", with equal variances under the null.

Test B I have constructed myself. Here, the null distribution used is an asymmetric generic discrete distribution. But I have found the following comment in Rohatgi & Saleh (2001, 2nd ed, p. 462)

"If the distribution is not symmetric, the $p$-value is not well defined in the two-sided case, although many authors recommend doubling the one-sided $p$-value".

The authors do not discuss this further, nor do they comment on the "many authors suggestion" to double the one-sided $p$-value. (This creates the question "double the $p$-value of which side? And why this side and not the other?)

I was not able to find any other comment, opinion or result on this whole matter. I understand that with an asymmetric distribution although we can consider an interval symmetric around the null hypothesis as regards the value of the parameter, we will not have the second usual symmetry, that of probability mass allocation. But I do not understand why this makes the $p$-value "not-well defined". Personally, by using an interval symmetric around the null hypothesis for the values of the estimator I see no definitional problem in saying "the probability that the null distribution will produce values equal to the boundaries of, or outside this interval is XX". The fact that the probability mass on the one side will be different than the probability mass on the other side, does not appear to cause troubles, at least for my purposes. But it is rather more probable than not that Rohatgi & Saleh know something that I don't.

So this is my question: In what sense the $p$-value is (or can be) "not well defined" in the case of a two-sided test when the null distribution is not symmetric?

A perhaps important note: I approach the matter more in a Fisherian spirit, I am not trying to obtain a strict decision rule in the Neyman-Pearson sense. I leave it up to the user of the test to use the $p$-value information alongside any other information to make inferences.

$\endgroup$
3
  • 5
    $\begingroup$ In addition to the likelihood-based ("Fisherian") and LR-based (N-P) approaches, another method considers how to obtain short confidence intervals and uses those for hypothesis testing. This is done in the spirit of decision theory (and using its methods), where length is included within the loss function. For unimodal symmetric distributions of the test statistic, obviously the shortest possible intervals are obtained using symmetric intervals (essentially "doubling the p-value" of one-sided tests). Shortest-length intervals depend on the parameterization: thus they cannot be Fisherian. $\endgroup$
    – whuber
    Mar 6, 2015 at 16:30
  • $\begingroup$ I was wondering if the answers posted here would be also applicable on beta distributions. Thanks. $\endgroup$
    – JLT
    Jul 13, 2017 at 10:40
  • $\begingroup$ @JLT: Yes, why not? $\endgroup$ Jul 13, 2017 at 11:27

2 Answers 2

15
$\begingroup$

If we look at the 2x2 exact test, and take that to be our approach, what's "more extreme" might be directly measured by 'lower likelihood'. (Agresti[1] mentions a number of approaches by various authors to computing two tailed p-values just for this case of the 2x2 Fisher exact test, of which this approach is one of the three specifically discussed as 'most popular'.)

For a continuous (unimodal) distribution, you just find the point in the other tail with the same density as your sample value, and everything with equal or lower likelihood in the the other tail is counted in your computation of p-value.

For discrete distributions which are monotonically nonincreasing in the tails, it's just about as simple. You just count everything with equal or lower likelihood than your sample, which given the assumptions I added (to make the term "tails" fit with the idea), gives a way to work it out.

If you're familiar with HPD intervals (and again, we're dealing with unimodality), it's basically like taking everything outside an open HPD interval that's bounded in one tail by your sample statistic.

enter image description here

[To reiterate -- this is likelihood under the null we're equating here.]

So at least in the unimodal case, it seems simple enough to emulate Fisher's exact test and still talk about the two tails.

However, you may not have intended to invoke the spirit of Fisher's exact test in quite this way.

So thinking outside that idea of what makes something 'as, or more extreme' for a moment, let's head just slightly more toward the Neyman-Pearson end of things. It can help (before you test!) to set about defining a rejection region for a test conducted at some generic level $\alpha$ (I don't mean you have to literally compute one, just how you would compute one). As soon as you do, the way to compute two tailed p-values for your case should become obvious.

This approach can be valuable even if one is conducting a test outside the usual likelihood ratio test. For some applications, it can be tricky to figure out how to compute p-values in asymmetric permutation tests... but it often becomes substantially simpler if you think about a rejection rule first.

With F-tests of variance, I've noticed that the "double one tail p-value" can give quite different p-values to what I see as the right approach. [I insist, for example, that it shouldn't matter which group you call "sample 1", or whether you put the larger or the smaller variance in the numerator - yet with some common approaches these apparently reasonable conditions are violated.]

[1]: Agresti, A. (1992),
A Survey of Exact Inference for Contingency Tables
Statistical Science, Vol. 7, No. 1. (Feb.), pp. 131-153.

$\endgroup$
16
  • 1
    $\begingroup$ ctd... If we're doing a likelihood ratio test, the likelihood ratio is always one-tailed, but if we construct an equivalent two tailed test based on some statistic then we still look to smaller likelihood ratios to locate "more extreme" $\endgroup$
    – Glen_b
    Mar 4, 2015 at 11:45
  • 2
    $\begingroup$ Alecos, what @Glen_b described is definitely closer to my intuitive understanding of the word "extreme", than what you are describing in the last two comments. Imagine a unimodal but super-skewed distribution of the test statistic, so that $-0.1$ is a very very uncommon value under the null, but $+0.1$ is still very common. If you obtain the value $-0.1$ in some dataset, this should certainly cast doubt on the null, but under your interpretation you would look at $|-0.1|=0.1$ which is not suspicious at all. By taking absolute value you destroy the sensitivity! $\endgroup$
    – amoeba
    Mar 4, 2015 at 12:41
  • 1
    $\begingroup$ @amoeba Certainly, sometimes you have to write it in order to see it. Indeed, we are in the land of probabilities so "extreme in terms of probability" appears to me too to be the right approach than "extreme in terms of mathematical value". And only the asymmetric setup helps to clarify that, in the symmetric case the distinction is lost, at least for the "no-effect" null hypothesis. One more case textbooks do not prepare you for... $\endgroup$ Mar 4, 2015 at 12:51
  • 2
    $\begingroup$ Doubling the one-tailed p-value might be defended as a Bonferroni correction for carrying out two one-tailed tests. After all, following a two-tailed test, we're usually very much inclined to regard any doubt cast on the truth of the null as favouring another hypothesis whose direction is determined by the data. $\endgroup$ Mar 4, 2015 at 13:20
  • 2
    $\begingroup$ Ultimately, yes. The neat thing about Fisher's approach is it gives a very sensible way of arriving at a p-value without even having an alternative. But if you do have specific alternatives of interest, you can target your rejection region more or less precisely to those alternatives by declaring the parts of the sample space where the alternatives will tend to put your samples as rejection region. A test statistic, T, is a convenient way of achieving that, in essence by associating a single number with each point in it (giving us a 'more extreme' as measured by T). ... ctd $\endgroup$
    – Glen_b
    Mar 4, 2015 at 23:22
14
$\begingroup$

A p-value's well-defined once you create a test statistic that partitions the sample space & orders the partitions according to your notions of increasing discrepancy with the null hypothesis. (Or, equivalently, once you create a set of nested rejection regions of decreasing size.) So what R. & S. are getting at is that if you consider either high or low values of a statistic $S$ to be interestingly discrepant with your null hypothesis you still have a little work to do to get a proper test statistic $T$ from it. When $S$ has a symmetric distribution around nought they seem to leap to $T=|S|$ without much thought, & therefore regard the asymmetric case as presenting a puzzle.

Doubling the lowest one-tailed p-value can be seen as a multiple-comparisons correction for carrying out two one-tailed tests. After all, following a two-tailed test, we're usually very much inclined to regard any doubt cast on the truth of the null as favouring another hypothesis whose direction is determined by the observed data. A proper test statistic is then $t=\min(\Pr_{H_0}(S<s),\Pr_{H_0}(S>s))$, & when $S$ has a continuous distribution the p-value is given by $2t$.

When $S$ has a continuous distribution, the approach to forming a two-tailed test shown by @Glen_b—defining the density of $S$ as the test statistic: $T=f_S(S)$—will of course produce valid p-values; but I'm not sure that it was ever recommended by Fisher, or that it's currently recommended by neo-Fisherians. If at first glance it appears more principled somehow than doubling the one-tailed p-value, note that having to deal with probability density rather than mass means that that the two-tailed p-value thus calculated may change when the test-statistic is transformed by an order-preserving function. For example, if to test the null that a Gaussian mean is equal to nought, you take a single observation $X$ & obtain $1.66$, the value with equal density at the other tail is $-1.66$, & the p-value therefore $$p=\Pr(X > 1.66) +\Pr(X<-1.66)=0.048457+0.048457=0.09691.$$ But if you consider it as testing the null that a log-Gaussian geometric mean is equal to one & take a single observation $Y$ & obtain $\mathrm{e}^{1.66}=5.2593$, the value with equal density at the other tail is $0.025732$($=\mathrm{e}^{-3.66}$), & the p-value therefore $$p=\Pr(Y>5.2593) +\Pr(Y<0.025732)=0.048457+0.00012611=0.04858.$$

enter image description here

Note that cumulative distribution functions are invariant to order-preserving transformations, so in the example above doubling the lowest p-value gives \begin{align}p=2t&=2\min(\Pr(X<1.66),\Pr(X>1.66))\\&=2\min(\Pr(Y<5.2593),\Pr(Y>5.2593))\\&=2\min(0.048457,0.951543)\\&=2\times 0.048457=0.09691.\end{align}

A kind of sequel to this answer, discussing some principles of test construction in which the alternative hypothesis is explicitly stated, can be found here.

† When $S$ has a discrete distribution, writing

$$p_\mathrm{L} = \Pr_{H_0}(S\leq s)$$ $$p_\mathrm{U} = \Pr_{H_0}(S\geq s)$$

for the lower & upper one-tailed p-values, the two-tailed p-value is given by

$$ \Pr(T\leq t) = \begin{cases} p_\mathrm{L} + \Pr_{H_0}(P_\mathrm{U} \leq p_\mathrm{L}) & \text{when}\ p_\mathrm{L} \leq p_\mathrm{U}\\ p_\mathrm{U} + \Pr_{H_0}(P_\mathrm{L} \leq p_\mathrm{U}) & \text{otherwise} \end{cases} $$

; i.e. by adding to the smaller one-tailed p-value the largest achievable p-value in the other tail that does not exceed it. Note that $2t$ is still an upper bound.

$\endgroup$
16
  • 1
    $\begingroup$ Oh wow. This is a very good point, +1. What is your advice then? Also, can I interpret this discrepancy as corresponding to different (in this case implicit) choices of test statistic? $\endgroup$
    – amoeba
    Mar 5, 2015 at 18:05
  • 1
    $\begingroup$ @amoeba: Not a typo! And when you observe 1.66 you take the the minimum of 0.952 & 0.048. If you actually observed -3.66 it'd be the minimum of 0.0001 & 0.9999. $\endgroup$ Mar 6, 2015 at 16:40
  • 1
    $\begingroup$ @Scortchi I have just accepted Glen_b's answer because it was more "useful" to me in the narrow sense. But yours helped me to avoid the trap of thinking that "that's all there is to it", which is an excellent insurance policy for future risks. Thanks again. $\endgroup$ Mar 20, 2015 at 18:39
  • 1
    $\begingroup$ @Scortchi I have to agree; my response took a rather simplistic and one-sided view, and I should qualify, extend and justify the answer. I'll probably do that in several stages. $\endgroup$
    – Glen_b
    Mar 22, 2015 at 6:06
  • 1
    $\begingroup$ @Glen_b: Thanks, I look forward to it. I also want to extend mine to show how score tests & generalized likelihood ratio tests give different answers (in general); & the theory of unbiased tests is surely worth mentioning in this context (but I can barely remember it). $\endgroup$ Mar 22, 2015 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.