4
$\begingroup$

For a two-sided hypothesis test, we need to sum both tails when calculating the p-value. For a test statistic that has a continuous and symmetric distribution like the two-sample t-test, this is quite trivial. It is less trivial (to me at least) for a test statistic with a discrete, asymmetric distribution. Take the binomial test for example. Say our null hypothesis is that the probability of Bernoulli success, $p=0.4$. When I use the binom.test in R, I get the following results:

binom.test(5,10,.4)

meaning I saw 5 heads out of 10 trials, the p-value it returns is:

$$\phi = 1-{10\choose 4}.4^4.6^6-{10 \choose 3}.4^3.6^7=0.5342$$ In other words, from the 5 binomial terms, the ones corresponding to 3 and 4 successes were excluded from the p-value.

Now, what if I observe 3 successes:

binom.test(3,10,.4)

The p-value now returned is: $$\phi = 1-{10\choose 4}.4^4.6^6=0.7492$$ So now, only the term corresponding to 4 successes was excluded from the p-value.

What is the concrete rule to say which terms should be excluded in the two-sided test? Equivalently, is it possible for anyone to point to the actual source code on GitHub? I couldn't locate it.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ You can inspect the code simply by typing binom.test in the console - the relevant part is the assignment of a value to the PVAL variable. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 16, 2020 at 15:53

1 Answer 1

Reset to default
3
$\begingroup$

I think I got the answer on the mechanical level by searching the scipy repo (the function performs equivalently to the R function): https://github.com/scipy/scipy/search?q=binom_test&unscoped_q=binom_test and locating binom_test. They basically take the terms on the other side of the tail which have a PMF lower than the input, $x$. In the example given in the question, the PMF of the binomial at $5$ is lower than the one at $3$. Hence, the term corresponding to $5$ is included in the p_Value when $x=3$ but the term corresponding to $3$ is not included in the p_Value when $x=5$. Still looking for some intuition for this.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ (+1) Perhaps edit your q. or ask a new one about why the two-tailed p-value should be defined this way. Especially unintuitive is that this p-value isn't a bimonotone function of the probability postulated by the null hypothesis. For 3 successes in 10 trials, increasing $p$ from 0.40 to 0.41 lowers the p-value, as you'd expect, from 0.7492 to 0.5409; but increasing it to only 0.408 raises the p-value to 0.7495 $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 16, 2020 at 13:09
  • $\begingroup$ @Scortchi-ReinstateMonica - doesn't it make sense in that for the p-value, we want the probability of observing something as or more extreme than the null in the direction of the alternate? And your observation is an unfortunate consequence when applying this principal to discrete distributions? $\endgroup$
    – ryu576
    Mar 23, 2020 at 2:54
  • $\begingroup$ Makes sense, but there's still the question of how you should define "more extreme" in the first place. Rather than looking at probability under the null, why not e.g. the ratio of probability under the null to probability under the most favourable alternative? I think you're right about the unfortunate consequence: the only way I know to ensure a sensible relation between p-values & null hypothesis is to take the simple approach of doubling the lesser one-tailed p-value; which does mean chucking out the simple definition of a p-value. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 23, 2020 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.