How to calculate the p-value of a test, that checked for a binary property? 16081 subjects were randomly assigned into two groups:


*

*test group: 7916 subjects

*control group: 8165 subjects


Only the test group was exposed to something. During the test period 


*

*10 subjects of the test group

*8 subjects of the control group


showed a specific behavior (only true or false possible).
How can I calculate the p-value for this result, i.e., the probability of the exposition not having any effect on the occurrence of this behavior?

My current attempt (in Python 3) looks as follows:
from scipy.stats import chisquare

test_group_size = 7916
test_group_true = 10
test_group_false = test_group_size - test_group_true

control_group_size = 8165
control_group_true = 8
control_group_false = control_group_size - control_group_true

expected_true = control_group_true * test_group_size / control_group_size
expected_false = control_group_false * test_group_size / control_group_size

_, p_value = chisquare([test_group_true, test_group_false],
                       f_exp=[expected_true, expected_false])

print(p_value)

The output is
0.4201628893079947

But is this correct?

It differs quite a lot from the 0.296, that these two websites output:


*

*abtestguide -> result

*vwo -> result
Abtestguide allows to choose between one-sided and two-sided, but the p-value does not change with this choice:


*

*one-sided

*two-sided
Might this be a bug in the code of the website?

From looking at the source code of one of them, I re-created the result in python:
from scipy.stats import norm
import numpy as np

test_group_size = 7916
test_group_true = 10

control_group_size = 8165
control_group_true = 8

true_rate_a = control_group_true / control_group_size
true_rate_b = test_group_true / test_group_size

se_a_sq = (true_rate_a * (1 - true_rate_a)) / control_group_size
se_b_sq = (true_rate_b * (1 - true_rate_b)) / test_group_size

se_diff = np.sqrt(se_a_sq + se_b_sq)
zScore = (true_rate_b - true_rate_a) / se_diff

p_value = 1 - norm.cdf(zScore, 0, 1)
print(p_value)

output:
0.2958346408590914

 A: Arrange the results as a $2\times 2$-table and use chi2_contigency from SciPy in Python to obtain the correct $p$-value (here shown without continuity correction):
import numpy as np
from scipy.stats import chi2_contingency, fisher_exact

obs = np.array([[8157, 8],[7906,10]])

g, p, dof, expctd = chi2_contingency(obs, correction = False)

p

0.59094761107842753

So the $p$-value is roughly $0.5909$.
A viable alternative would be to use Fisher's exact test. This can be done using fisher_exact from SciPy:
oddsr, p_fish = fisher_exact(obs)

oddsr

1.289685049329623

p_fish

0.64294290970149048

The odds ratio is $1.29$ with an associated $p$-value from Fisher's exact test of $0.643$.
A: The main problem here seems to be a confusion over one and two sided tests. The value of 0.64 is for a two-sided test (confirmed using R) but the websites are doing one-sided tests although I get 0.38 for that which I suspect is due to a difference in how the web-sites treat the $p$-value for the obtained table.
A: It looks like you could treat this as a 2x2 contingency table and use Fisher's Exact Test.
So, I'd recommend you use scipy.stats.chi2_contingency instead of chisquare.
