2
$\begingroup$

From what I understand from a pervious question I asked, the p-value for a particular binomial test (specifying a number of flips, and number of heads and tails, and a null-hypothesis to test against) loosely describes the probability that this data is expected with the assumption that the null-hypothesis is correct.

So if I do a binomial test for binom_test(numberOfHeads=9, numberOfFlips=10, weight = .5), and obtain a pvalue for the 2-tailed test. This represents the probability that such an event (nine heads in 10 flips) could occur by a fair coin. If this number is very small, then we can assume that the null hypothesis is rejected with some level of statistical confidence.

My issue is that I'm having a lot of trouble understanding what the derivation of the binomial test. In python there is a binomial test function, but they simply link to the wikipedia article on binomial test.

Intuitively if my null hypothesis is p=1/2, shouldn't my binomial test just be the binomial distribution? For example, I would have thought that my pvalue getting 4 heads in 5 flips would be just the probability of a fair coin getting 4-heads in five flips. So ${5 \choose 4} (1/2)^4(1/2)^5$. Is this correct? I am looking at what is done on wikipedia and it doesn't seem as though it's in agreement with my thoughts - but what they have written (particularly for the two tailed test) is very unclear to me.

$\endgroup$
1
  • $\begingroup$ Here is a related post. The p-value is the probability of the observed result or something more extreme. In your last example with $y=4$ heads you should calculate the probability of observing 4 out of 5 heads OR 5 out of 5 heads. This would be the upper-tailed probability using the binomial CDF, $1-F_Y(4-1,n,p)$. This is the one-sided upper p-value. $\endgroup$ Nov 7, 2021 at 22:50

4 Answers 4

3
$\begingroup$

shouldn't my binomial test just be the binomial distribution

Beware speaking so loosely that concepts become muddled.

A test is not just a statistic (e.g. it also needs data, hypotheses and a rejection rule for example) and a statistic is not a distribution (though it has a distribution). Given the assumptions, the test statistic will have the null distribution (a binomial with success probability $\frac12$) when the null is true, and some other distribution (a binomial with a different success probability if the assumptions otherwise hold) when it is false.

This represents the probability that such an event (nine heads in 10 flips) could occur by a fair coin

Again your difficulty appears to be speaking too loosely and confusing yourself - or perhaps instead you have not seen it correctly defined; in any case, it is not correct. Rather the $p$ value here is "the probability of an outcome at least as extreme" as $9$ heads (given $H_0$); the cases you count here are $0, 1, 9$ or $10$ heads, all the cases at least as far from half the cases being heads as $9$ heads is.

would have thought that my pvalue getting 4 heads in 5 flips would be just the probability of a fair coin getting 4-heads in five flips. So ${5\choose 4} (\frac12)^4(\frac12)^5$. Is this correct?

No, for the same reason. With a two tailed test the cases that are at least as extreme as $4$ heads for $5$ tosses are $0, 1, 4$ and $5$ heads.

[Note also that - since the test statistic is discrete - we cannot achieve just any type I error rate we choose, so the use of $p$ values without considering the available significance levels can sometimes be misleading; I often see people compare their $p$ values to $0.05$ without even checking to see whether they can even get below $0.05$. For the $10$-toss case the available two tailed significance levels anywhere near typical significance levels are about $0.2\%$, $2.15\%$ and $10.94\%$, so a rejection rule of "reject when p≤0.05" would correspond to $\alpha=2.15\%$. For the 5-tosses case there's no two tailed significance level below $6.25\%$. Choose your rejection rules with care, or you might have a test that can never reject the null.]

loosely describes the probability that this data is expected with the assumption that the null-hypothesis is correct.

This is so loose as to be potentially misleading. It's easy to lead yourself into misinterpreting that phrasing; it's only correctly interpreted when you will take that to mean something equivalent to "the probability of a test statistic at least as extreme as the one from our sample, under $H_0$".

$\endgroup$
6
  • $\begingroup$ Thanks for the answer. I still am a bit perplexed though: I understand now that p-value here is defined as "the probability of an outcome at least as extreme" - but I am very lost as to the intuition why we would do this rather than just using the "the probability of an outcome" on its own. Like why not just reject events which have probabilities below a certain value? $\endgroup$ Nov 8, 2021 at 1:23
  • $\begingroup$ If the probability of 97 heads out of 100 flips is extremely small, why can't I just use that probability to judge if the hypothesis is correct? Why instead am I now counting this value: P(H=98)+P(H=99)+P(H=100)+P(H=0)+P(H=1)+P(H=2)? That is, why is it more useful for me to work with this sum rather than just P(H=97)? What is it about these events being "as or more extreme" than P(H=97) make it relevant when we consider the specific outcome of an event with exactly 97 heads? $\endgroup$ Nov 8, 2021 at 1:28
  • $\begingroup$ Let's say your cutoff for "very small" is some specific value -- e.g. 1 in 100, 1 in 1000. Then there's some sample size at which you'll reject every possible outcome as unlikely, even the ones most consistent with the null. e.g. with 6400 coin tosses, even eve center value at p=1/2 has probability less than 1 in 100. With a continuous response you have this problem at every sample size. It requires more thought. $\endgroup$
    – Glen_b
    Nov 8, 2021 at 1:50
  • $\begingroup$ This is why I like coming to hypothesis testing purely from the rejection rule approach; if you ask yourself about which sorts of observations (/which values of test statistics based on the observations) would in general terms lead you to want to reject the null, and which ones would not, in almost all practical situations you are led immediately to sensible tests, and all that is left is figuring out where to put the boundary of the rejection region. With coin tossing, if you get a proportion of heads close to 1/2 you're less inclined to reject the null than if the proportion is far from 1/2. $\endgroup$
    – Glen_b
    Nov 8, 2021 at 1:57
  • $\begingroup$ This immediately then leads to "reject for cases where the proportion is not close to 1/2" -- and we then just need to figure out where our "just not close enough" boundary is, which - at some significance level and sample size, will be straightforward to find. In some cases the issue is a little subtler (sometimes we need to think more carefully about how to define "more extreme", but there are good ways to do that). $\endgroup$
    – Glen_b
    Nov 8, 2021 at 2:01
1
$\begingroup$

There are two kinds of tests for binomial proportions. One uses a normal approximation to binomial distributions. It works fine if $n$ is large enough and $p$ is sufficiently near $1/2$ (roughly speaking, so that $np$ and $n(1-p)$ both exceed $5).$

Example: Suppose you have $x = 45$ Successes out of $n = 50$ Bernoulli trials. You wish to test $H_0: p= 0.5$ against $H_a: p \ne 0.5.$ Then $p$ is estimated as $\hat p = 45/50 = 0.9.$ Clearly, $0.9$ is different from $0.5,$ but the question is whether the difference is large enough to be significant based on $n = 50$ trials.

Approximate normal test. The test statistic is $z = \frac{\hat p-.5} { \sqrt\frac{ (.5)(.5)}{50} } = 5.657.$

z = (.9-.5)/sqrt(.25/50);  z
[1] 5.656854

Because a standard normal random variable $Z$ has $P(!Z|\ge 1.96) = 0.05,$ we reject $H_0$ at the 5% level, if our observed $z$ has $|z|\ge 1.96.$ Because our observed $z = 5.657,$ we reject $H_0.$

The P-value of the test is the probability of a more extreme value of $z$ than we observed: $P(|z| \ge 5.656854) \approx 0.$ So, if the true value of $p$ is $1/2,$ then it is almost impossible to get an observed $\hat p$ so far from $1/2$ (in either direction).

2*pnorm(-5.656854)
[1] 1.541728e-08

An exact binomial test uses suitable values $L$ and $U$ such that $X\sim\mathsf{Binom}(n=50, p=.5)$ has $P(X \le L)+P(X \ge U)$ just barely smaller than $0.05.$

Finding the quantiles $.025$ and $.975$ of $\mathsf{Binom}(50, .5)$ is a good start.

qbinom(c(.025,.975), 50, .5)
[1] 18 32

Then with some experimentation in R, we can find that $L =17$ and $U=33,$ so that $P(X \le L)+P(X \ge U) = 0.0328.$ [Because of the discreteness of the binomial distribution, we can't get closer to $5\%$ without going over.]

pbinom(17,50,.5)
[1] 0.01641957
sum(dbinom(33:50,50,.5))
[1] 0.01641957

This exact binomial test is called binom.test in R. [Notes: (1) The null hypothesis $H_0: p = 0.5$ is the default and so need not be specified in the code. (2) Output of the test is a P-value, so the critical values $L$ and $U$ need not be shown. One rejects at the 5% level, if the P-value is smaller than 5%.]

binom.test(45,50)

        Exact binomial test

data:  45 and 50
number of successes = 45, number of trials = 50, 
p-value = 4.21e-09
alternative hypothesis: 
 true probability of success is not equal to 0.5
95 percent confidence interval:
 0.7818646 0.9667249
sample estimates:
probability of success 
                   0.9 

Here is a plot of the PDF of $\mathsf{Binom}(n=50, p=0.5).$ The approximating normal density curve is shown (black) line, and the critical values $L$ and $U$ for the exact binomial test are shown as vertical dotted red lines.

enter image description here

Your example with $x = 9$ Successes in $n = 10$ trials, does not meet the criterion for an approximate normal test. So here is the exact binomial test. $H_0$ is rejected because the P-value $0.02148 < 0.05 = 5\%.$

binom.test(9, 10)

        Exact binomial test

data:  9 and 10
number of successes = 9, number of trials = 10, 
 p-value = 0.02148
alternative hypothesis: 
 true probability of success is not equal to 0.5
95 percent confidence interval:
 0.5549839 0.9974714
sample estimates:
probability of success 
                   0.9 

Notes: (1) The approximate normal test, with critical values $\pm 1.96$ for $z,$ may seem superficially to be exactly at the 5% level. However, values of $z$ very close to $\pm 1.96$ cannot always be achieved in reality.

(2) Some of the technical language in your question is vague, confusing, or incorrect. I see that @Glen-b (+1) has taken the trouble to comment on that in his Answer, so I will not do so here. A good plan would be to read his Answer before mine, and again after.

(3) Here is R code for the figure:

x = 0:50;  PDF=dbinom(x, 50, .5)
plot(x, PDF, type="h", lwd=2, col="blue", 
  main= "BINOM(50, .5) with Normal Approx")
 abline(h=0, col="green2")
 abline(v=0, col="green2")
 abline(v=c(17.5,32.5), col="red", lty="dotted")
 curve(dnorm(x, 25, sqrt(50/4)), add=T)
$\endgroup$
1
  • $\begingroup$ Thanks, I added an answer to try to explain what I learned and what was confusing. I'm not at 100% understanding, but many things have certainly been clarified. $\endgroup$ Nov 8, 2021 at 14:27
0
$\begingroup$

Thanks to @BruceET and @Glen_b some confusion was clarified, which I figured I'd write here to be helpful for anyone who also had my issue.

The main issue that I had was that I previously did a simulation to look at the frequency at which fair coins produce p-values that would be false-positives.

If I have a p-value of 3.4% and test it for p-value<3.5% (some interval that compensates for the discrete nature), this 3.4% doesn't mean that there is a 3.4% chance of reproducing the same data, but instead is that there is a 3.4% chance that a fair coin with that data would produce a p-value of 3.4%.

What was hard for me to understand, after getting that idea, is that this p-value seems like just an arbitrary function, and we could've used any function and characterized the chance said data would produce a certain output for that function.

But, from what I'm gathering, the utility of the p-value is that it typically adds a weight to all unlikely events. So you're sort of heuristically collecting all probabilities that are as and more unlikely in the parameter of interest and adding them up. Not quite at 100% understanding (and please correct me in the comments), but I think I am at least more correct than I was.

$\endgroup$
1
  • $\begingroup$ If you're doing a test here, then you have a null hypothesis that specifies a particular numerical value $p_0$ for the success probability $p.$ Data lead to an estimated value $\hat p$ of $p.$ Then given the data, the P-value is not just an "arbitrary function". It is the probability (assuming the null hypothesis to be true) of getting an estimated value farther from $p_0$ than $\hat p.$ Your conclusion (whether to believe $p_0$ is plausible) is based on the only information you have: the data. $\endgroup$
    – BruceET
    Nov 8, 2021 at 16:39
0
$\begingroup$

There are already some very good answers here, but I figured I would offer my own take.

You're correct in identifying the interpretation of the p value of the test: The probability of seeing 9 or more heads from a fair coin assuming all other assumptions are correct.

Note that the binomial test is in terms of the coin's bias, not in terms of the number of heads as you might expect for the binomial distribution. But, I can always obtain the number of heads by multiplying by the number of flips made since $\widehat{p} = x/n$, where $p$ is the estimated bias, $x$ is the observed number of heads, and $n$ is the number of flips.

Because $n\widehat{p}$ is an estimate of the expectation of the number of heads, it has some desirable properties. Due to the central limit theorem, we know

$$ n\widehat{p} \sim \operatorname{Normal}\big(np_0, np(1-p_0)\big) $$

where $p_0$ is the true bias. This implies

$$ \dfrac{\widehat{p} - p_0}{\sqrt{\dfrac{\widehat{p}(1-\widehat{p})}{n}}}\sim \operatorname{Normal}\big(0, 1\big) $$

Here, I've replaced $p_0$ with $\widehat{p}$ in the expression for the variance so align with the common Wald test statistic. Keeping $p_0$ in the variance expression would result in a Score test.

This is the test statistic for the binomial test. Our test statistic is based off a normal approximation to the binomial distribution, which becomes better when $n \to \infty$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.