9
$\begingroup$

When the null hypothesis is simple (i.e., has only distribution of the sample) and the only distribution of the sample is continuous, the p-value can be shown to be uniformly distributed over (0,1).

When the null hypothesis is composite (i.e., has more than one sample distributions), is the p-value still uniformly distributed over (0,1)? Why?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ In many situations with a composite null, the distribution of p-values depends on the true state of nature. The case of testing the mean of a Normal distribution with unknown mean and unknown SD provides a standard example. $\endgroup$
    – whuber ♦
    Commented May 14, 2013 at 0:07

2 Answers 2

Reset to default
4
$\begingroup$

The answer seems to be 'No' but sometimes 'Yes, asymptotically, depending on the construction of p'. Robins, van der Vaart, and Ventura (2000) go through the details.

The reasoning is basically what @whuber states telegraphically in the comment. The state of nature matters to the nuisance parameters.

Improve this answer
$\endgroup$
4
$\begingroup$

First of all let's deal with the simple point hypothesis. It is often claimed (as it is currently in Wikipedia) that the p-value is uniform over [0,1].

This is obviously not the case for discrete outcomes. For example if a coin is tossed 5 times and the test statistic is the number of heads obtained there are only 3 possible p-values under the null hypothesis that the coin is fair. The p-values 0.0625, 0.375 and 1 have probability 0.0625, 0.3125 and 0.625 of occuring arising from outcomes {0,5}, {1,4} and {2,3} respectively.

Perhaps less obvious is that it is not always true for the continuous case. Consider the the null hypothesis that the location of the minute hand on my clock when the battery runs out is uniformly distributed over the range [0,60). All outcomes have the same probability of occuring so the probability of getting an outcome as extreme or more extreme than the observed outcome is unity.

Moving on to composite nulls. No the p-value is not distributed over (0,1) for composite nulls and not just for the reasons given above. Perhaps the p-value has no meaning at all for composite nulls.

Alternatively the p-value is taken to be the p-value for the most favourable member of the composite null for the observed data i.e. the maximum likelihood member of the null. Under this interpretation a p-value of zero may not be possible. Consider the null hypothesis that a coin is fair where fair is defined with a bit of tolerance so that 0.49 < p < 0.51. If zero heads are observed the most likely point hypothesis is p=0.49. But using this member of the null there are more extreme outcomes: in this case all heads would be even less probable. Hence there is a lower bound on the p-value which is not 0.

Note that considering the least favourable member of the null for a given dataset doesn't get you anywhere - for some composite hypotheses it will always be possible to choose a member of the null for which the probability of the observed data is zero whatever the data may be (and therefore a p-value which is always zero). An example is the null that the coin is biased where biased is defined as outside of the above range i.e. p<0.49 or p>0.51. If some heads are observed the point hypothesis p=0 renders this impossible. Conversely if no heads are obtained p=1 renders this impossible. Therefore p-value cannot be defined in terms of least favourable member of the null.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I don't understand: "All outcomes have the same probability of occuring so the probability of getting an outcome as extreme or more extreme than the observed outcome is unity." What is the ordering on the circle that makes an event more extreme? $\endgroup$
    – Horst Grünbusch
    Commented Aug 13, 2014 at 9:12
  • $\begingroup$ Outcomes as extreme count towards the p-value. So that's my point: there is no particular ordering making an event more extreme or less extreme. So I am not sure how to test my hypothesis under the Fisher paradigm without specificying an alternative. (Actually I have some alternatives in mind: one is that the minute hand is more likely to stop at 12 o'clock because my clock chimes on the hour which typically kills the battery. Another is that I have a very heavy minute hand that takes more energy to lift - so it is more likely to stop on the upward cycle i.e. on the left half of the clock) $\endgroup$
    – spiderdancer
    Commented Aug 13, 2014 at 14:01
  • 2
    $\begingroup$ Obviously the alternative distribution is any distribution different from $unif(S^1)$ and we agree that we cannot order these distributions in terms of e.g. a location parameter as this would be our proper habit in the euclidean, normal world. But from this I till don't see evidence that in circular statistics, $p$-values for testing valid point hypotheses in a continuous family of distributions may not be uniformly distributed. $\endgroup$
    – Horst Grünbusch
    Commented Aug 13, 2014 at 17:06
  • 2
    $\begingroup$ In my reading of the clock situation (with "times" in $(-\pi,\pi]$), it appears @Horst is correct. But first a particular alternative hypothesis must be specified. Perhaps (say) it is the non-uniform von Mises distributions of mean $\mu+\pi$. By the Neyman-Pearson Lemma, a UMP critical region for a test of size $\alpha$ is the interval $[-\alpha\pi,\alpha\pi]$. By construction, the chance of the observation lying in this region equals $100\alpha\%$--not unity. $\endgroup$
    – whuber ♦
    Commented Oct 13, 2014 at 18:55
  • 1
    $\begingroup$ Wikipedia says: p-value is a key concept in the approach of Ronald Fisher, where he uses it to measure the weight of the data against a specified hypothesis... Fisher's approach does not involve any alternative hypothesis, which is instead a feature of the Neyman–Pearson approach.' The point I was trying to show with the clock example is that sometimes it is possible to devise a 'bad' test statistic where the use of 'more extreme' in the definition of p-value is problematic. Admittedly other test statistics may be available for the same null but the choice of statistic can be arbitrary. $\endgroup$
    – spiderdancer
    Commented Oct 14, 2014 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.