# Checking independence of A/B/C/… test results using Python

Assume I collected the following results from an A/B/C/D/E test:

        Result 1    Result 2    Result 3
Control 21          7           32
VAR1    65          25          64
VAR2    107         50          94
VAR3    15          8           15
VAR4    189         6           2


The null hypothesis is that the results are independent and I want to use Chi-Squared Test of Independence [1]. To that end, I would like to use scipy.stats.chi2_contingency. However, in order to better understand the happenings, I reproduce the computations manually, as described in [1].

When taking the whole results set, everything works as expected; the chi2 test statistic and the p-value computed manually or using scipy.stats.chi2_contingency are identical. This is also the case when taking subsets of the results (e.g. only two columns OR only two variations). I get puzzled once taking a 2x2 subset of the results (equivalent to a binary A/B test). Once doing so, the numbers that I get manually are no longer the same as those returned by scipy.stats.chi2_contingency.

Here is a summary of the code behind my exercise:

import pandas as pd
import numpy as np
from scipy import  stats

def comp_expected(df):
"""
Compute the expected matrix
"""
return pd.DataFrame(
np.outer(df.sum(axis=1), df.sum(axis=0)) / df.sum().sum(),
columns=df.columns,
index=df.index
)

chi_squared_stat = lambda observed, expected: (((observed-expected)**2)/expected).sum().sum()
p_value = lambda observed, chi_squared_stat: 1 - stats.chi2.cdf(
x=chi_squared_stat,
df=(observed.shape[0]-1)*(observed.shape[1]-1))

def compare_computations(observed):
print(observed)
chi2_stat = chi_squared_stat(observed, comp_expected(observed))
pval = p_value(observed, chi_squared_stat(observed, comp_expected(observed)))
chi2_contingency = stats.chi2_contingency(observed)
print('chi2_stat = {}\npval = {}\nchi2_contingency output = {}'.format(
chi2_stat,
pval,
chi2_contingency
))
print("chi2_stat identical - {}".format(np.isclose(chi2_stat, chi2_contingency[0])))
print("pval      identical - {}".format(np.isclose(pval, chi2_contingency[1])))

# Observed data
observed_base = pd.DataFrame(
{
"Result 1": [21, 65, 107, 15, 189],
"Result 2": [7, 25, 50, 8, 6],
"Result 3": [32, 64, 94, 15, 2]
},
index=["Control",
"VAR1",
"VAR2",
"VAR3",
"VAR4"]
)


Now you can compare the results of compare_computations(observed_base) and compare_computations(observed_base.iloc[0:2][observed_base.columns[0:2]]). The full example can be found here [2].

# Question(s):

• Why in the latter case the "manual" computation does not yield the same results as the one provided by scipy.stats.chi2_contingency?
• As far as I understand, if the returned p-value is smaller then 0.05, it means that the null hypothesis can be rejected with 95% certainty. Is that correct?
• What about the statistical power of the results? How can I compute it?

The answer to the first question is in the documentation of scipy.stats.chi2_contingency. In particular, in case the number of degrees of freedom is 1, then "Yates’ correction for continuity" is applied.
By replacing stats.chi2_contingency(observed) with stats.chi2_contingency(observed, correction=False) the results of the manual computation and the ones provided by chi2_contingency are always consistent.