# how to calculate p-value based on # of trials and observed values?

In this article, the p-value is calculated as 0.32 for observing 55 heads in 100 coin flips.

https://netflixtechblog.com/interpreting-a-b-test-results-false-positives-and-statistical-significance-c1522d0db27a

"In the example of observing 55 heads in 100 coin flips, we calculated a p-value of 0.32. Because the p-value is larger than the 0.05 significance level, we conclude that there is not statistically significant evidence that the coin is unfair."

However, when i try this using scipy library, here's what i get:

from scipy import stats

n = 100  # Number of coin flips
p = 0.5  # Probability of getting heads with a fair coin

# Calculate the p-value
p_value = 1 - stats.binom.cdf(observed_heads - 1, n, p)
print("P-value:", p_value)

P-value: 0.18410080866334777


How do they calculate p-value of 0.32?

You are conducting a one-sided test, which discounts $$\le 45$$ as evidence against the null of fairness. This is the appropriate hypothesis for testing that there is a bias in favor of heads.

The Netflixers are doing a two-sided test, so seeing too few heads is also in the direction of this alternative hypothesis. This is a test of fairness, so it will be sensitive to a bias in either direction:

. bitesti 100 55 0.5, detail

Binomial probability test

N   Observed k   Expected k   Assumed p   Observed p
----------------------------------------------------------------
100           55           50     0.50000      0.55000

Pr(k >= 55)            = 0.184101  (one-sided test)
Pr(k <= 55)            = 0.864373  (one-sided test)
Pr(k <= 45 or k >= 55) = 0.368202  (two-sided test)

Pr(k == 55)            = 0.048474  (observed)
Pr(k == 46)            = 0.057958
Pr(k == 45)            = 0.048474  (opposite extreme)


Generally, outcomes with $$k$$ successes are considered “as extreme or more extreme” than the observed outcome of 55 if $$\Pr(k) \le \Pr(55) = 0.048474$$.

The difference between 0.368202 and 0.32 is most likely due to using an approximate t-test of means instead of the exact binomial, which gives a p-value of $$\approx 0.32$$.

• from the article, "were we to repeat, many times, the experiment of flipping a coin 100 times and calculating the fraction of heads, with a fair coin (the null hypothesis is true), in 32% of those experiments the outcome would feature at least 55% heads or at least 55% tails (results at least as unlikely as our actual observation).". shouldn't this be 55% heads and 45% tails?
– kms
Commented Oct 19, 2023 at 19:44
• I think 55+ heads and <=45 heads are the extreme regions. The latter is equivalent to 55+ tails, so the article quote seems correct. Commented Oct 20, 2023 at 1:38