0
$\begingroup$

Assume I collected the following results from an A/B/C/D/E test:

        Result 1    Result 2    Result 3
Control 21          7           32
VAR1    65          25          64
VAR2    107         50          94
VAR3    15          8           15
VAR4    189         6           2

The null hypothesis is that the results are independent and I want to use Chi-Squared Test of Independence [1]. To that end, I would like to use scipy.stats.chi2_contingency. However, in order to better understand the happenings, I reproduce the computations manually, as described in [1].

When taking the whole results set, everything works as expected; the chi2 test statistic and the p-value computed manually or using scipy.stats.chi2_contingency are identical. This is also the case when taking subsets of the results (e.g. only two columns OR only two variations). I get puzzled once taking a 2x2 subset of the results (equivalent to a binary A/B test). Once doing so, the numbers that I get manually are no longer the same as those returned by scipy.stats.chi2_contingency.

Here is a summary of the code behind my exercise:

import pandas as pd
import numpy as np
from scipy import  stats


def comp_expected(df):
    """
    Compute the expected matrix
    """
    return pd.DataFrame(
        np.outer(df.sum(axis=1), df.sum(axis=0)) / df.sum().sum(),
        columns=df.columns,
        index=df.index
    )


chi_squared_stat = lambda observed, expected: (((observed-expected)**2)/expected).sum().sum()
p_value = lambda observed, chi_squared_stat: 1 - stats.chi2.cdf(
    x=chi_squared_stat, 
    df=(observed.shape[0]-1)*(observed.shape[1]-1))

def compare_computations(observed):
    print(observed)
    chi2_stat = chi_squared_stat(observed, comp_expected(observed))
    pval = p_value(observed, chi_squared_stat(observed, comp_expected(observed)))
    chi2_contingency = stats.chi2_contingency(observed)
    print('chi2_stat = {}\npval = {}\nchi2_contingency output = {}'.format(
        chi2_stat, 
        pval,
        chi2_contingency
    ))
    print("chi2_stat identical - {}".format(np.isclose(chi2_stat, chi2_contingency[0])))
    print("pval      identical - {}".format(np.isclose(pval, chi2_contingency[1])))

# Observed data
observed_base = pd.DataFrame(
    {
        "Result 1": [21, 65, 107, 15, 189],
        "Result 2": [7, 25, 50, 8, 6],
        "Result 3": [32, 64, 94, 15, 2]
    },
    index=["Control", 
           "VAR1", 
           "VAR2", 
           "VAR3", 
           "VAR4"]
)

Now you can compare the results of compare_computations(observed_base) and compare_computations(observed_base.iloc[0:2][observed_base.columns[0:2]]). The full example can be found here [2].

Question(s):

  • Why in the latter case the "manual" computation does not yield the same results as the one provided by scipy.stats.chi2_contingency?
  • As far as I understand, if the returned p-value is smaller then 0.05, it means that the null hypothesis can be rejected with 95% certainty. Is that correct?
  • What about the statistical power of the results? How can I compute it?
$\endgroup$
0
$\begingroup$

The answer to the first question is in the documentation of scipy.stats.chi2_contingency. In particular, in case the number of degrees of freedom is 1, then "Yates’ correction for continuity" is applied.

By replacing stats.chi2_contingency(observed) with stats.chi2_contingency(observed, correction=False) the results of the manual computation and the ones provided by chi2_contingency are always consistent.

I am still not sure about the other two questions. Now, in addition, I have to investigate what Yates' correction is.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.