# How to calculate the p-value of a test, that checked for a binary property?

16081 subjects were randomly assigned into two groups:

• test group: 7916 subjects
• control group: 8165 subjects

Only the test group was exposed to something. During the test period

• 10 subjects of the test group
• 8 subjects of the control group

showed a specific behavior (only true or false possible).

How can I calculate the p-value for this result, i.e., the probability of the exposition not having any effect on the occurrence of this behavior?

My current attempt (in Python 3) looks as follows:

from scipy.stats import chisquare

test_group_size = 7916
test_group_true = 10
test_group_false = test_group_size - test_group_true

control_group_size = 8165
control_group_true = 8
control_group_false = control_group_size - control_group_true

expected_true = control_group_true * test_group_size / control_group_size
expected_false = control_group_false * test_group_size / control_group_size

_, p_value = chisquare([test_group_true, test_group_false],
f_exp=[expected_true, expected_false])

print(p_value)


The output is

0.4201628893079947


But is this correct?

It differs quite a lot from the 0.296, that these two websites output:

Abtestguide allows to choose between one-sided and two-sided, but the p-value does not change with this choice:

Might this be a bug in the code of the website?

From looking at the source code of one of them, I re-created the result in python:

from scipy.stats import norm
import numpy as np

test_group_size = 7916
test_group_true = 10

control_group_size = 8165
control_group_true = 8

true_rate_a = control_group_true / control_group_size
true_rate_b = test_group_true / test_group_size

se_a_sq = (true_rate_a * (1 - true_rate_a)) / control_group_size
se_b_sq = (true_rate_b * (1 - true_rate_b)) / test_group_size

se_diff = np.sqrt(se_a_sq + se_b_sq)
zScore = (true_rate_b - true_rate_a) / se_diff

p_value = 1 - norm.cdf(zScore, 0, 1)
print(p_value)


output:

0.2958346408590914


Arrange the results as a $$2\times 2$$-table and use chi2_contigency from SciPy in Python to obtain the correct $$p$$-value (here shown without continuity correction):

import numpy as np
from scipy.stats import chi2_contingency, fisher_exact

obs = np.array([[8157, 8],[7906,10]])

g, p, dof, expctd = chi2_contingency(obs, correction = False)

p

0.59094761107842753


So the $$p$$-value is roughly $$0.5909$$.

A viable alternative would be to use Fisher's exact test. This can be done using fisher_exact from SciPy:

oddsr, p_fish = fisher_exact(obs)

oddsr

1.289685049329623

p_fish

0.64294290970149048


The odds ratio is $$1.29$$ with an associated $$p$$-value from Fisher's exact test of $$0.643$$.

• Interesting, thanks! I just found these two websites: abtestguide and vwo, which should basically do the same. However they both output 0.296 with my values (result from abtestguide, result from vwo). Any idea on why these values are so much different from yours? (I'll add these links to my question too.) Commented Nov 8, 2018 at 12:58
• @TobiasHermann The results are different because these websites apply a one-sided test. Note that the $p$-value of $0.295$ is exactly half of mine $0.59$. Commented Nov 8, 2018 at 13:26
• Mhh, but the p-value on this website does not change, no matter if I choose one-sided or two-sided. Is this a bug in their code? Commented Nov 8, 2018 at 14:32
• @TobiasHermann Probably, I don't know. But the $p$-value from Python is definitely correct (for a two-sided test). Commented Nov 8, 2018 at 14:55

The main problem here seems to be a confusion over one and two sided tests. The value of 0.64 is for a two-sided test (confirmed using R) but the websites are doing one-sided tests although I get 0.38 for that which I suspect is due to a difference in how the web-sites treat the $$p$$-value for the obtained table.

It looks like you could treat this as a 2x2 contingency table and use Fisher's Exact Test.

So, I'd recommend you use scipy.stats.chi2_contingency instead of chisquare.

• COOLSerdash beat me to it. :-) Commented Nov 8, 2018 at 10:55