18
$\begingroup$

My situation is as follows: I want, through a Monte-Carlo study, to compare $p$-values of two different tests for statistical significance of an estimated parameter (null is "no effect - parameter is zero", and the implied alternative is "parameter is not zero"). Test A is the standard "independent two-sample t-test for equality of means", with equal variances under the null.

Test B I have constructed myself. Here, the null distribution used is an asymmetric generic discrete distribution. But I have found the following comment in Rohatgi & Saleh (2001, 2nd ed, p. 462)

"If the distribution is not symmetric, the $p$-value is not well defined in the two-sided case, although many authors recommend doubling the one-sided $p$-value".

The authors do not discuss this further, nor do they comment on the "many authors suggestion" to double the one-sided $p$-value. (This creates the question "double the $p$-value of which side? And why this side and not the other?)

I was not able to find any other comment, opinion or result on this whole matter. I understand that with an asymmetric distribution although we can consider an interval symmetric around the null hypothesis as regards the value of the parameter, we will not have the second usual symmetry, that of probability mass allocation. But I do not understand why this makes the $p$-value "not-well defined". Personally, by using an interval symmetric around the null hypothesis for the values of the estimator I see no definitional problem in saying "the probability that the null distribution will produce values equal to the boundaries of, or outside this interval is XX". The fact that the probability mass on the one side will be different than the probability mass on the other side, does not appear to cause troubles, at least for my purposes. But it is rather more probable than not that Rohatgi & Saleh know something that I don't.

So this is my question: In what sense the $p$-value is (or can be) "not well defined" in the case of a two-sided test when the null distribution is not symmetric?

A perhaps important note: I approach the matter more in a Fisherian spirit, I am not trying to obtain a strict decision rule in the Neyman-Pearson sense. I leave it up to the user of the test to use the $p$-value information alongside any other information to make inferences.

$\endgroup$
  • 4
    $\begingroup$ In addition to the likelihood-based ("Fisherian") and LR-based (N-P) approaches, another method considers how to obtain short confidence intervals and uses those for hypothesis testing. This is done in the spirit of decision theory (and using its methods), where length is included within the loss function. For unimodal symmetric distributions of the test statistic, obviously the shortest possible intervals are obtained using symmetric intervals (essentially "doubling the p-value" of one-sided tests). Shortest-length intervals depend on the parameterization: thus they cannot be Fisherian. $\endgroup$ – whuber Mar 6 '15 at 16:30
  • $\begingroup$ I was wondering if the answers posted here would be also applicable on beta distributions. Thanks. $\endgroup$ – JLT Jul 13 '17 at 10:40
  • $\begingroup$ @JLT: Yes, why not? $\endgroup$ – Scortchi Jul 13 '17 at 11:27
11
$\begingroup$

If we look at the 2x2 exact test, and take that to be our approach, what's "more extreme" might be directly measured by 'lower likelihood'. (Agresti[1] mentions a number of approaches by various authors to computing two tailed p-values just for this case of the 2x2 Fisher exact test, of which this approach is one of the three specifically discussed as 'most popular'.)

For a continuous (unimodal) distribution, you just find the point in the other tail with the same density as your sample value, and everything with equal or lower likelihood in the the other tail is counted in your computation of p-value.

For discrete distributions which are monotonically nonincreasing in the tails, it's just about as simple. You just count everything with equal or lower likelihood than your sample, which given the assumptions I added (to make the term "tails" fit with the idea), gives a way to work it out.

If you're familiar with HPD intervals (and again, we're dealing with unimodality), it's basically like taking everything outside an open HPD interval that's bounded in one tail by your sample statistic.

enter image description here

[To reiterate -- this is likelihood under the null we're equating here.]

So at least in the unimodal case, it seems simple enough to emulate Fisher's exact test and still talk about the two tails.

However, you may not have intended to invoke the spirit of Fisher's exact test in quite this way.

So thinking outside that idea of what makes something 'as, or more extreme' for a moment, let's head just slightly more toward the Neyman-Pearson end of things. It can help (before you test!) to set about defining a rejection region for a test conducted at some generic level $\alpha$ (I don't mean you have to literally compute one, just how you would compute one). As soon as you do, the way to compute two tailed p-values for your case should become obvious.

This approach can be valuable even if one is conducting a test outside the usual likelihood ratio test. For some applications, it can be tricky to figure out how to compute p-values in asymmetric permutation tests... but it often becomes substantially simpler if you think about a rejection rule first.

With F-tests of variance, I've noticed that the "double one tail p-value" can give quite different p-values to what I see as the right approach. [It shouldn't matter which group you call "sample 1", or whether you put the larger or the smaller variance in the numerator.]

[1]: Agresti, A. (1992),
A Survey of Exact Inference for Contingency Tables
Statistical Science, Vol. 7, No. 1. (Feb.), pp. 131-153.

$\endgroup$
  • 1
    $\begingroup$ ctd... If we're doing a likelihood ratio test, the likelihood ratio is always one-tailed, but if we construct an equivalent two tailed test based on some statistic then we still look to smaller likelihood ratios to locate "more extreme" $\endgroup$ – Glen_b Mar 4 '15 at 11:45
  • 2
    $\begingroup$ Doubling the one-tailed p-value might be defended as a Bonferroni correction for carrying out two one-tailed tests. After all, following a two-tailed test, we're usually very much inclined to regard any doubt cast on the truth of the null as favouring another hypothesis whose direction is determined by the data. $\endgroup$ – Scortchi Mar 4 '15 at 13:20
  • 1
    $\begingroup$ @Alecos it's simple enough to justify a symmetric choice! I find it hard to see how you'd read what I wrote as suggesting a symmetric choice was in any way not a valid thing to do (that choice is covered by the discussion I gave about the rejection rule - you can easily construct a symmetric rejection rule). The first part of my answer was responding to the part in the question about Fisher. If you ask about Fisher, should I not discuss what it seems Fisher might do, based on what he did in similar circumstances? You seem to interpret my response as saying more than it is. $\endgroup$ – Glen_b Mar 4 '15 at 14:42
  • 1
    $\begingroup$ @Alecos In particular, I am not advocating Fisher, or Neyman Pearson approaches (whether we are talking about likelihood ratio tests or just hypothesis tests more generally), nor should you consider me as trying to suggest that anything that I have omitted might be wrong. I'm just discussing a number of the things you seemed to be raising in your question. $\endgroup$ – Glen_b Mar 4 '15 at 14:51
  • 2
    $\begingroup$ Ultimately, yes. The neat thing about Fisher's approach is it gives a very sensible way of arriving at a p-value without even having an alternative. But if you do have specific alternatives of interest, you can target your rejection region more or less precisely to those alternatives by declaring the parts of the sample space where the alternatives will tend to put your samples as rejection region. A test statistic, T, is a convenient way of achieving that, in essence by associating a single number with each point in it (giving us a 'more extreme' as measured by T). ... ctd $\endgroup$ – Glen_b Mar 4 '15 at 23:22
8
$\begingroup$

A p-value's well-defined once you create a test statistic that partitions the sample space & orders the partitions according to your notions of increasing discrepancy with the null hypothesis. (Or, equivalently, once you create a set of nested rejection regions of decreasing size.) So what R. & S. are getting at is that if you consider either high or low values of a statistic $S$ to be interestingly discrepant with your null hypothesis you still have a little work to do to get a proper test statistic $T$ from it. When $S$ has a symmetric distribution around nought they seem to leap to $T=|S|$ without much thought, & therefore regard the asymmetric case as presenting a puzzle.

Doubling the lowest one-tailed p-value can be seen as a multiple-comparisons correction for carrying out two one-tailed tests. After all, following a two-tailed test, we're usually very much inclined to regard any doubt cast on the truth of the null as favouring another hypothesis whose direction is determined by the observed data. A proper test statistic is then $t=\min(\Pr_{H_0}(S<s),\Pr_{H_0}(S>s))$, & when $S$ has a continuous distribution the p-value is given by $2t$.

When $S$ has a continuous distribution, the approach to forming a two-tailed test shown by @Glen_b—defining the density of $S$ as the test statistic: $T=f_S(S)$—will of course produce valid p-values; but I'm not sure that it was ever recommended by Fisher, or that it's currently recommended by neo-Fisherians. If at first glance it appears more principled somehow than doubling the one-tailed p-value, note that having to deal with probability density rather than mass means that that the two-tailed p-value thus calculated may change when the test-statistic is transformed by an order-preserving function. For example, if to test the null that a Gaussian mean is equal to nought, you take a single observation $X$ & obtain $1.66$, the value with equal density at the other tail is $-1.66$, & the p-value therefore $$p=\Pr(X > 1.66) +\Pr(X<-1.66)=0.048457+0.048457=0.09691.$$ But if you consider it as testing the null that a log-Gaussian geometric mean is equal to one & take a single observation $Y$ & obtain $\mathrm{e}^{1.66}=5.2593$, the value with equal density at the other tail is $0.025732$($=\mathrm{e}^{-3.66}$), & the p-value therefore $$p=\Pr(Y>5.2593) +\Pr(Y<0.025732)=0.048457+0.00012611=0.04858.$$

enter image description here

Note that cumulative distribution functions are invariant to order-preserving transformations, so in the example above doubling the lowest p-value gives \begin{align}p=2t&=2\min(\Pr(X<1.66),\Pr(X>1.66))\\&=2\min(\Pr(Y<5.2593),\Pr(Y>5.2593))\\&=2\min(0.048457,0.951543)\\&=2\times 0.048457=0.09691.\end{align}

A kind of sequel to this answer, discussing some principles of test construction in which the alternative hypothesis is explicitly stated, can be found here.

† When $S$ has a discrete distribution, writing

$$p_\mathrm{L} = \Pr_{H_0}(S\leq s)$$ $$p_\mathrm{U} = \Pr_{H_0}(S\geq s)$$

for the lower & upper one-tailed p-values, the two-tailed p-value is given by

$$ \Pr(T\leq t) = \begin{cases} p_\mathrm{L} + \Pr_{H_0}(P_\mathrm{U} \leq p_\mathrm{L}) & \text{when}\ p_\mathrm{L} \leq p_\mathrm{U}\\ p_\mathrm{U} + \Pr_{H_0}(P_\mathrm{L} \leq p_\mathrm{U}) & \text{otherwise} \end{cases} $$

; i.e. by adding to the smaller one-tailed p-value the largest achievable p-value in the other tail that does not exceed it. Note that $2t$ is still an upper bound.

$\endgroup$
  • 1
    $\begingroup$ Oh wow. This is a very good point, +1. What is your advice then? Also, can I interpret this discrepancy as corresponding to different (in this case implicit) choices of test statistic? $\endgroup$ – amoeba Mar 5 '15 at 18:05
  • 1
    $\begingroup$ @amoeba: Not a typo! And when you observe 1.66 you take the the minimum of 0.952 & 0.048. If you actually observed -3.66 it'd be the minimum of 0.0001 & 0.9999. $\endgroup$ – Scortchi Mar 6 '15 at 16:40
  • 1
    $\begingroup$ @Scortchi I have just accepted Glen_b's answer because it was more "useful" to me in the narrow sense. But yours helped me to avoid the trap of thinking that "that's all there is to it", which is an excellent insurance policy for future risks. Thanks again. $\endgroup$ – Alecos Papadopoulos Mar 20 '15 at 18:39
  • 1
    $\begingroup$ @Scortchi I have to agree; my response took a rather simplistic and one-sided view, and I should qualify, extend and justify the answer. I'll probably do that in several stages. $\endgroup$ – Glen_b Mar 22 '15 at 6:06
  • 1
    $\begingroup$ @Glen_b: Thanks, I look forward to it. I also want to extend mine to show how score tests & generalized likelihood ratio tests give different answers (in general); & the theory of unbiased tests is surely worth mentioning in this context (but I can barely remember it). $\endgroup$ – Scortchi Mar 22 '15 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.