# Assessing A/B test results using Python

## Problem's setting

Assume having results of an A/B test. You let you users experience two variants of your website and you counted how many converted:

|        | Converted | Total visits |
|--------|-----------|--------------|
|Control | $c_n$     | $c_t$        |
|Variant | $v_n$     | $v_t$        |


We have that the conversion rates are:

\begin{align*} \mathrm{CR}_c & = \frac{c_n}{c_t}\\ \mathrm{CR}_v & = \frac{v_n}{v_t} \end{align*}

We may pose three null hypotheses:

• Two sided/tailed: $H_0: \mathrm{CR}_c = \mathrm{CR}_v$ and thus the alternative hypothesis is: $H_a: \mathrm{CR}_c \neq \mathrm{CR}_v$
• One sided/tailed: $H_0: \mathrm{CR}_c > \mathrm{CR}_v$ (or $H_0: \mathrm{CR}_c < \mathrm{CR}_v$) and in turn the alternative hypothesis is $H_a: \mathrm{CR}_c <= \mathrm{CR}_v$ (or $H_a: \mathrm{CR}_c >= \mathrm{CR}_v$)

Before moving on, here is a list of some related discussions on Cross Validate:

## Two sided case

As can be learned from the linked discussions, in this case there are two possible tests that can exploited: $z$-test and $\chi^2$-test. Here is an example:

data = pd.DataFrame({
"Converted": [123, 133],
"Total": [231, 262]
}, index=['Control', 'Var'])

# Output:
# |         | Converted | Total
# | Control |   123       | 231
# | Var     |   133       | 262


Using statsmodels the two tests can be executed as follows:

ssp.proportions_ztest(count=data.Converted, nobs=data.Total, alternative='two-sided')

# Output:
# (0.5507210148592081, 0.5818249364318755)

ssp.proportions_chisquare(count=data.Converted, nobs=data.Total)

# Output:
# (0.30329363620755795, 0.5818249364318744, (array([[123, 108],
[133, 129]]), array([[119.95131846, 111.04868154],
[136.04868154, 125.95131846]])))


Using scipy.stats, the story is a little different. For the $\chi^2$-test, the input should be a contingency matrix which can be computed using pandas.DataFrame.concat:

scipy.stats.chi2_contingency(
pd.concat(
[data.Total - data.Converted, data.Converted],
axis=1),
correction=False)

# (0.303293636207558, 0.5818249364318744, 1, array([[111.04868154, 119.95131846],
[125.95131846, 136.04868154]]))


Note that the correction is set to False. Otherwise, the Yates’ correction kicks in.

It can easily be seen that the $p$-values are the same (up to some floating point operations differences). Furthermore, the square of the $z$ static is the same as the one of the $\chi^2$. In this specific case, it is clear that the resulting $p$-value are large, and thus it is impossible to reject the null hypothesis.

I am afraid I cannot point you to a straightforward $z$-test using scipy.

## One sided case

The one sided case can only be evaluated using the $z$-test, and thus statsmodels is used. For the sake of demonstration, I consider three results. First, the control and variation groups have almost the same conversion rate:

data = pd.DataFrame({
"Converted": [100, 145],
"Total": [200, 300]
}, index=['Control', 'Var'])

ssp.proportions_ztest(
count=data.Converted, nobs=data.Total, alternative='smaller'),
ssp.proportions_ztest(
count=data.Converted, nobs=data.Total, alternative='larger')

# ((0.3652214232606525, 0.642526936116749),
# (0.3652214232606525, 0.35747306388325095))


In this case, both one-sided tests yield high $p$-values; the null hypothesis cannot be rejected. Now, have a look in a case where the control set has a much lower conversion rate:

data = pd.DataFrame({
"Converted": [80, 145],
"Total": [200, 300]
}, index=['Control', 'Var'])

ssp.proportions_ztest(
count=data.Converted, nobs=data.Total, alternative='smaller'),
ssp.proportions_ztest(
count=data.Converted, nobs=data.Total, alternative='larger')

# ((-1.8349396085439338, 0.033257319037821406),
# (-1.8349396085439338, 0.9667426809621786))


Now, the first test (corresponding to the "smaller" case) yields a low $p$-value. As a matter of fact so low that we can say it is $\sim 97\%$ safe to reject the null hypothesis being that the conversion rate of the alternative treatment is smaller. The symmetric case is when the null hypothesis is that the conversion rate of the alternative treatment is larger than the one of the control group:

data = pd.DataFrame({
"Converted": [130, 145],
"Total": [200, 300]
}, index=['Control', 'Var'])

ssp.proportions_ztest(
count=data.Converted, nobs=data.Total, alternative='smaller'),
ssp.proportions_ztest(
count=data.Converted, nobs=data.Total, alternative='larger')

# ((3.669879217087869, 0.9998786674495203),
# (3.669879217087869, 0.00012133255047971392))


In this case the resulting $p$-values are going the other way around, suggesting that we can reject the "larger" null hypothesis. It is important to stress that you have to be careful what exactly you want to check, pick the suitable test and make sure you interpret the results accordingly.

## Questions

1. Is the problem setting I consider is a 2-sample or 1-sample? In particular in regards to the documentation of statsmodels.stats.power.NormalIndPower and statsmodels.stats.power.GofChisquarePower.
2. Is there a $z$-test for the setting I am considering using scipy?
3. Am I using the tests provided by statsmodels and scipy correctly for the problem setting I am considering? Should I take additional aspects into consideration?
4. Is there a way to run a one-sided $\chi^2$-test? As far as I understand it doesn't make sense.