How to calculate the p-value of a test, that checked for a binary property?

Question

16081 subjects were randomly assigned into two groups:

• test group: 7916 subjects
• control group: 8165 subjects

Only the test group was exposed to something. During the test period

• 10 subjects of the test group
• 8 subjects of the control group

showed a specific behavior (only true or false possible).

How can I calculate the p-value for this result, i.e., the probability of the exposition not having any effect on the occurrence of this behavior?

---

My current attempt (in Python 3) looks as follows:

from scipy.stats import chisquare

test_group_size = 7916
test_group_true = 10
test_group_false = test_group_size - test_group_true

control_group_size = 8165
control_group_true = 8
control_group_false = control_group_size - control_group_true

expected_true = control_group_true * test_group_size / control_group_size
expected_false = control_group_false * test_group_size / control_group_size

_, p_value = chisquare([test_group_true, test_group_false],
                       f_exp=[expected_true, expected_false])

print(p_value)

The output is

0.4201628893079947

But is this correct?

---

It differs quite a lot from the 0.296, that these two websites output:

• abtestguide -> result
• vwo -> result

Abtestguide allows to choose between one-sided and two-sided, but the p-value does not change with this choice:

• one-sided
• two-sided

Might this be a bug in the code of the website?

---

From looking at the source code of one of them, I re-created the result in python:

from scipy.stats import norm
import numpy as np

test_group_size = 7916
test_group_true = 10

control_group_size = 8165
control_group_true = 8

true_rate_a = control_group_true / control_group_size
true_rate_b = test_group_true / test_group_size

se_a_sq = (true_rate_a * (1 - true_rate_a)) / control_group_size
se_b_sq = (true_rate_b * (1 - true_rate_b)) / test_group_size

se_diff = np.sqrt(se_a_sq + se_b_sq)
zScore = (true_rate_b - true_rate_a) / se_diff

p_value = 1 - norm.cdf(zScore, 0, 1)
print(p_value)

output:

0.2958346408590914

Answer 1

Arrange the results as a $2\times 2$-table and use chi2_contigency from SciPy in Python to obtain the correct $p$-value (here shown without continuity correction):

import numpy as np
from scipy.stats import chi2_contingency, fisher_exact

obs = np.array([[8157, 8],[7906,10]])

g, p, dof, expctd = chi2_contingency(obs, correction = False)

p

0.59094761107842753

So the $p$-value is roughly $0.5909$.

A viable alternative would be to use Fisher's exact test. This can be done using fisher_exact from SciPy:

oddsr, p_fish = fisher_exact(obs)

oddsr

1.289685049329623

p_fish

0.64294290970149048

The odds ratio is $1.29$ with an associated $p$-value from Fisher's exact test of $0.643$.

Answer 2

The main problem here seems to be a confusion over one and two sided tests. The value of 0.64 is for a two-sided test (confirmed using R) but the websites are doing one-sided tests although I get 0.38 for that which I suspect is due to a difference in how the web-sites treat the $p$-value for the obtained table.

Answer 3

It looks like you could treat this as a 2x2 contingency table and use Fisher's Exact Test.

So, I'd recommend you use scipy.stats.chi2_contingency instead of chisquare.