Understanding p-value

I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification.

Here is the definition of p-value from Wikipedia:

The p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. (http://en.wikipedia.org/wiki/P-value)

My first question pertains to the expression "at least as extreme as the one that was actually observed." My understanding of the logic underlying the use of p-value is the following: If the p-value is small, it's unlikely that the observation occurred assuming the null hypothesis and we may need an alternative hypothesis to explain the observation. If the p-value is not so small, it is likely that the observation occurred only assuming the null hypothesis and the alternative hypothesis is not necessary to explain the observation. So if someone wants to insist on a hypothesis he/she has to show that the p-value of the null hypothesis is very small. With this view in mind, my understanding of the ambiguous expression is that p-value is $\min[P(X<x),P(x<X)]$, if the PDF of the statistic is unimodal, where $X$ is the test statistic and $x$ is its value obtained from the observation. Is this right? If it is right, is it still applicable to use the bimodal PDF of the statistic? If two peaks of the PDF are separated well and the observed value is somewhere in the low probability density region between the two peaks, which interval does the p-value give the probability of?

The second question is about another definition of p-value from Wolfram MathWorld:

The probability that a variate would assume a value greater than or equal to the observed value strictly by chance. (http://mathworld.wolfram.com/P-Value.html)

I understood that the phrase "strictly by chance" should be interpreted as "assuming a null hypothesis". Is that right?

The third question regards the use of "null hypothesis". Let's assume that someone wants to insist that a coin is fair. He expresses the hypothesis as that relative frequency of heads is 0.5. Then the null hypothesis is "relative frequency of heads is not 0.5." In this case, whereas calculating the p-value of the null hypothesis is difficult, the calculation is easy for the alternative hypothesis. Of course the problem can be resolved by interchanging the role of the two hypotheses. My question is that rejection or acceptance based directly on the p-value of the original alternative hypothesis (without introducing the null hypothesis) is whether it is OK or not. If it is not OK, what is usual workaround for such difficulties when calculating the p-value of a null hypothesis?

I posted a new question that is more clarified based on the discussion in this thread.

-
Of possible interest: Is there an error in the one-sided binomial test in R? – user10525 Nov 30 '12 at 13:17
You have caught a subtlety that often goes unrecognized: "more extreme" needs to be measured in terms of relative likelihood of the alternative hypothesis rather than in the obvious (but not generally correct) sense of being further out in the tail of the null sampling distribution. This is explicit in the formulation of the Neyman-Pearson Lemma, which is used to justify many hypothesis tests and to determine their critical regions (and whence their p-values). Thinking this through will help answer your first question. – whuber Nov 30 '12 at 14:33
As I recall, the Neyman-Pearson Lemma is optimal for simple vs. simple hypothesis tests (Ho: mu=mu_0, Ha: mu=mu_a). For composite tests (Ho: mu=mu_0, Ha: mu>mu_a) there is an alternative test. – RobertF Dec 3 '12 at 15:22
In the immortal words of David W. Hogg, "Holy shit, p-values are confusing!" – abaumann May 28 '13 at 16:39

You have to think at the concept of extreme in terms of probability of the test statistics, not in terms of its value or the value of the random variable being tested. I report the following example from Christensen, R. (2005). Testing Fisher, Neyman, Pearson, and Bayes. The American Statistician, 59(2), 121–126

$$\phantom{(r\;|\;\theta=0}r\; | \quad 1 \quad \quad 2 \quad \quad 3 \quad \quad 4\\ p(r\;|\;\theta=0) \; |\; 0.980\;0.005\; 0.005\; 0.010\\ \quad p\;\mathrm{value} \; \; | \;\; 1.0 \quad 0.01 \quad 0.01 \;\; 0.02$$

Here $r$ are the observations, the second line is the probability to observe a given observation under the null hypothesis $\theta=0$, that is used here as test statistics, the third line is the $p$ value. We are here in the framework of Fisherian test: there is one hypothesis ($H_0$, in this case $\theta=0$) under which we want to see whether the data are weird or not. The observations with the smallest probability are 2 and 3 with 0.5% each. If you obtain 2, for example, the probability to observe something as likely or less likely ($r=2$ and $r=3$) is 1%. The observation $r=4$ does not contribute to the $p$ value, although it's further away (if an order relation exists), because it has higher probability to be observed.

This definition works in general, as it accommodates both categorical and multidimensional variables, where an order relation is not defined. In the case of a ingle quantitative variable, where you observe some bias from the most likely result, it might make sense to compute the single tailed $p$ value, and consider only the observations that are on one side of the test statistics distribution.

I disagree entirely with this definition from Mathworld.

I have to say that I'm not completely sure I understood your question, but I'll try to give a few observations that might help you.

In the simplest context of Fisherian testing, where you only have the null hypothesis, this should be the status quo. This is because Fisherian testing works essentially by contradiction. So, in the case of the coin, unless you have reasons to think differently, you would assume it is fair, $H_0: \theta=0.5$. Then you compute the $p$ value for your data under $H_0$ and, if your $p$ value is below a predefined threshold, you reject the hypothesis (proof by contradiction). You never compute the probability of the null hypothesis.

With the Neyman-Pearson tests you specify two alternative hypotheses and, based on their relative likelihood and the dimensionality of the parameter vectors, you favour one or another. This can be seen, for example, in testing the hypothesis of biased vs. unbiased coin. Unbiased means fixing the parameter to $\theta=0.5$ (the dimensionality of this parameter space is zero), while biased can be any value $\theta \neq 0.5$ (dimensionality equal to one). This solves the problem of trying to contradict the hypothesis of bias by contradiction, which would be impossible, as explained by another user. Fisher and NP give similar results when the sample is large, but they are not exactly equivalent. Here below a simple code in R for a biased coin.

n <- 100  # trials
p_bias <- 0.45  # the coin is biased
k <- as.integer(p_bias * n)  # successes

# value obtained by plugging in the MLE of p, i.e. k/n = p_bias
lambda <- 2 * n * log(2) + 2 * k * log(p_bias) + 2 * (n-k) * log(1. - p_bias)

p_value_F <- 2 * pbinom(k, size=n, prob=0.5)  # p-value under Fisher test
p_value_NP <- 1 - pchisq(q=lambda, df=1)  # p-value under Neyman-Pearson
binom.test(c(k, n-k))  # equivalent to Fisher

-
+1 for pointing out a great article I didn't know about. (Also for some much needed skepticism about the utility of Mathworld's view of statistics). – conjugateprior Dec 1 '12 at 12:44
Thank you very much! So the p-value is \int_{x : f(x) <= k} f, where f is the PDF of a test statistic and k is the observed value of the statistic. Thank you again. – JDL Dec 1 '12 at 13:01
Regarding the third answer, what is proved in your answer is unfairness of the coin because fairness assumption is rejected. On the contrary, to prove fairness of the coin by contradiction, I have to assume unfairness \theta \neq 0.5 and calculate p-value of my data. How can I do it? My point is the difficulty originated from the \neq sign of the unfairness assumption. Do I have to introduce some tolerance level for fairness, say 0.4 < \theta < 0.6, and calculate p-value in terms of \theta and integrate it over 0 < \theta < 0.4 and 0.6 < \theta < 1 ? – JDL Dec 1 '12 at 13:02
One more question. This link explains "one-sided" p-value. It says one-sided p-value answers questions like "null hypothesis, that two populations really are the same ... what is the chance that randomly selected samples would have means as far apart as (or further than) observed in this experiment with the specified group having the larger mean?" Is it an appropriate use of one-sided p-value? I think the null hypothesis itself should be expressed as an inequality in this case (instead of equality and one-sided test). – JDL Dec 1 '12 at 15:05
@Zag, I disagree rather with this answer: you don't have to think of the concept of extreme in terms of probability. Better to say that in this example the probability under the null is being used as the test statistic - but that's not mandatory. For example, if the likelihood ratio, as mentioned by whuber, is used as a test statistic, it will not in general put possible samples in the same order as will probability under the null. Other statistics are chosen for maximum power against a specified alternative, or all alternatives, or for high power against a vaguely defined set. – Scortchi Dec 1 '12 at 16:20

(1) A statistic is a number you can calculate from a sample. It's used to put into order all the samples you might have got (under an assumed model, where coins don't land on their edges & what have you). If $t$ is what you calculate from the sample you actually got, & $T$ is the corresponding random variable, then the p-value is given by $\newcommand{\pr}{\mathrm{Pr}} \pr\left(T\geq t\right)$ under the null hypothesis, $H_0$. 'Greater than' vs 'more extreme' is unimportant in principle. For a two-sided test on a Normal mean we could use $\pr(|Z|\geq |z|)$ but it's convenient to use $2\min [\pr(Z\geq z),\pr(Z\leq z)]$ because we have the appropriate tables. (Note the doubling.)

There's no requirement for the test statistic to put the samples in order of their probability under the null hypothesis. There are situations (like Zag's example) where any other way would seem perverse (without more information about what $r$ measures, what kinds of discrepancies with $H_0$ are of most interest, &c.), but often other criteria are used. So you could have a bimodal PDF for the test statistic & still test $H_0$ using the formula above.

(2) Yes, they mean under $H_0$.

(3) A null hypothesis like "The frequency of heads is not 0.5" is no use because you would never be able to reject it. It's a composite null including "the frequency of heads is 0.49999999", or as close as you like. Whether you think beforehand the coin's fair or not, you pick a useful null hypothesis that bears on the problem. Perhaps more useful after the experiment is to calculate a confidence interval for the frequency of heads that shows you either it's clearly not a fair coin, or it's close enough to fair, or you need to do more trials to find out.

An illustration for (1):

Suppose you're testing the fairness of a coin with 10 tosses. There are $2^{10}$ possible results. Here are three of them:

$\mathsf{HHHHHHHHHH}\\ \mathsf{HTHTHTHTHT}\\ \mathsf{HHTHHHTTTH}$

You'll probably agree with me that the first two look a bit suspicious. Yet the probabilities under the null are equal:

$\mathrm{Pr}(\mathsf{HHHHHHHHHH}) = \frac{1}{1024}\\ \mathrm{Pr}(\mathsf{HTHTHTHTHT}) = \frac{1}{1024}\\ \mathrm{Pr}(\mathsf{HHTHHHTTTH}) = \frac{1}{1024}$

To get anywhere you need to consider what types of alternative to the null you want to test. If you're prepared to assume independence of each toss under both null & alternative (& in real situations this often means working very hard to ensure experimental trials are independent), you can use the total count of heads as a test statistic without losing information. (Partitioning the sample space in this way is another important job that statistics do.)

So you have a count between 0 and 10

t<-c(0:10)


Its distribution under the null is

p.null<-dbinom(t,10,0.5)


Under the version of the alternative that best fits the data, if you see (say) 3 out of 10 heads the probability of heads is $\frac{3}{10}$, so

p.alt<-dbinom(t,10,t/10)


Take the ratio of the probability under the null to the probability under the alternative (called the likelihood ratio):

lr<-p.alt/p.null


Compare with

plot(log(lr),p.null)


So for this null, the two statistics order samples the same way. If you repeat with a null of 0.85 (i.e. testing that the long-run frequency of heads is 85%), they don't.

p.null<-dbinom(t,10,0.85)
plot(log(lr),p.null)


To see why

plot(t,p.alt)


Some values of $t$ are less probable under the alternative, & the likelihood ratio test statistic takes this into account. NB this test statistic will not be extreme for

$\mathsf{HTHTHTHTHT}$

And that's fine - every sample can be considered extreme from some point of view. You choose the test statistic according to what kind of discrepancy to the null you want to be able to detect.

... Continuing this train of thought, you can define a statistic that partitions the sample space differently to test the same null against the alternative that one coin toss influences the next one. Call the number of runs $r$, so that

$\mathsf{HHTHHHTTTH}$

has $r=6$:

$\mathsf{HH}\ \mathsf{T}\ \mathsf{HHH}\ \mathsf{TTT}\ \mathsf{H}$

The suspicious sequence

$\mathsf{HTHTHTHTHT}$

has $r=10$. So does

$\mathsf{THTHTHTHTH}$

while at the other extreme

$\mathsf{HHHHHHHHHH}\\ \mathsf{TTTTTTTTTT}$

have $r=1$. Using probability under the null as the test statistic (the way you like) you can say that the p-value of the sample

$\mathsf{HTHTHTHTHT}$

is therefore $\frac{4}{1024}=\frac{1}{256}$. What's worthy of note, comparing this test to the previous, is that even if you stick strictly to the ordering given by probability under the null, the way in which you define your test statistic to partition the sample space is dependent on consideration of alternatives.

-
You say that the definition Pr(T \ge t; H_0) can be applicable to any multimodal (of course, including bimodal) PDF of a test statistic. Then, you and Zag give different p-values for multimodal PDF of a test statistic. IMHO, Zag's definition is more resonable because the role of p-value is to quantify how likely (or weird) the observation is under the null hypothesis, as he pointed. What is your rationale for the definition Pr(T \ge t; H_0) ? – JDL Dec 2 '12 at 14:41
@JDL, that just is the definition of a p-value. The question then becomes how to find a 'good' test statistic (& how to define 'good'). Sometimes the probability under the null (or any function of the data that gives the same ordering) is used as the test statistic. Sometimes there are good reasons to choose others, which fill up a lot of space in books on theoretical statistics. I think it's fair to say they involve explicit or implicit consideration of alternatives. ... – Scortchi Dec 3 '12 at 14:25
@JDL, ... And if a particular observation has low probability under both null & alternative it seems reasonable not to regard it as extreme. – Scortchi Dec 3 '12 at 14:27
Thank you for your answers, @Scortchi. I posted a new question and have seen your comments just now after the posting. Anyway, I'm still not clear about the definition. Thank you again for your kindly answers. – JDL Dec 3 '12 at 14:47
I added an illustration – Scortchi Dec 3 '12 at 15:54