# How to calculate statistical significance of an A/B test?

I am running an A/B/C test where users are shown three different introductions for a subscription service and they can then choose to receive weekly, monthly, or no content. I am tracking how many people see each option, and then from each group how many people choose each of the 3 options. I am trying to calculate the statistical significance of this. I am reading this article:

and I follow, except I'm not sure if my sample size is the full group of users or if it is just three, since I have 3 data points for each option. If the latter is the case, is three data points enough to work with? Is this the right path to be going down? Thanks!

• Is each person seeing all three ads or is there only one person seeing each ad? Am I correct in that you are trying to understand if (a) there is a statistically significant difference between those who chose the different subscriptions and which ad they say and (b) the p-value of the finding? Aug 11, 2020 at 17:04
• Every person only sees one of the three ads, but there is about 1000 people that see each of the three, so 3000 people total. You are correct though, I'd like to understand the statistical significance between which ad they see and which subscription option they choose. I'm not sure about the p-value, I read that 0.05 is a good value as a general rule. Aug 11, 2020 at 17:24

Thanks to @StatsStudent for the well-targeted question in a Comment and for your answer.

Below are simulated data for three ads, with 1000 subjects seeing each ad. Each subject makes a choice for 1 = Weekly, 2 = Monthly, or 3 = No. The preference vectors (parameter p in the sample procedure of R) show slight differences in prevalence in the population. For example, vector c(11, 11, 10) for group B amounts to probabilities $$11/32,\, 11/32,\, 10/32$$ for respective options $$1, 2, 3.$$

The question is whether the slightly different preferences, presumably due to seeing different ads, result in count differences among the three groups that rise to the level of statistical significance.

Data and contingency table. Here are simulation results of counts in the three groups, and the resulting contingency table of counts.

set.seed(811)  # for reproducibility
a = sample(1:3, 1000, rep=T, p=c(10,10,12))
A = tabulate(a); A
 326 319 355
b = sample(1:3, 1000, rep=T, p=c(11,11,10))
B = tabulate(b); B
 348 350 302
c = sample(1:3, 1000, rep=T, p=c(12,10,10))
C = tabulate(c); C
 387 310 303
MAT = rbind(A,B,C);  MAT
[,1] [,2] [,3]
A  326  319  355
B  348  350  302
C  387  310  303


Null hypothesis and expected counts. The null hypothesis is that the choices $$1,2,3$$ were made independently of which ad A, B, C was seen. The total counts in row A is, of course, 1000; the total count in column 1 is 1061; the grand total is $$n=3000.$$ If the null hypothesis is true we would expect that the number subjects seeing ad A and making choice 1 would be $$P(A)P(1) = P(A\cap 1)$$ estimated as follows $$\hat P(A) = 1000/3000, \hat P(1) = 1061/3000$$ so that the expected count for ad A and choice 1 is

$$E_{A1} = E_{11} = n\hat P(A)\hat P(1) = \frac{1000(1061)}{3000} = 353.6667,$$

Expected counts corresponding to the remaining eight cells of the table are computed similarly. [For the computations below, expected counts should not be rounded to integers.]

Chi-squared test. The test statistic for the chi-squared test of independence is

$$Q = \sum_{i=1}^3\sum_{j=1}^3 \frac{(X_{ij} - E_{ij})^2}{E_{ij}},$$ where $$X_{ij}$$ is the observed count in cell $$(i,j)$$ of the contingency matrix.

The first of the nine components of $$Q$$ is $$C_{11} = \frac{(326 - 353.67)^2}{353.67}=2.165.$$

Under the null hypothesis, $$Q \stackrel{aprx}{\sim}\mathsf{Chisq}(\nu),$$ where the 'degrees of freedom' $$\nu = (r-1)(c-1) = 4,$$ where $$r$$ and $$c$$ are the number of rows and columns, respectively, of the contingency matrix. [Notice that given the row and column totals and the four expected counts $$X_{11},X_{12},X_{21}, X_{22},$$ the remaining five $$X_{ij}$$ could be determined.]

This approximately chi-squared distribution is sufficiently accurate to give reliable results, provided that all of the expected counts $$E_{ij} > 5,$$ which is easily true for our data.

If $$Q > c = 9.488,$$ then we can reject the null hypothesis that Choices 1,2,3 are independent of Ads A,B,C at the 5% level of significance. The critical value $$c$$ cuts probability $$0.05 = 5\%$$ from the upper tail of $$\mathsf{Chisq}(4).$$

qchisq(.95, 4)
 9.487729


Chi-squared test in R. Below we show results of this chi-squared test of independence, computed in R. $$Q = 13.839 > 9.488.$$ so we reject at the 5% level of significance.

out = chisq.test(MAT); out

Pearson's Chi-squared test

data:  MAT
X-squared = 13.839, df = 4, p-value = 0.007826


The P-value is $$P(Q > 13.839),$$ computed according to $$\mathsf{Chisq}(4).$$ (The slight discrepancy from the computation in R below is that the output rounds $$Q,$$ labeled as X-squared.

1 - pchisq(13.839, 4)
 0.007827032


Additional information on intermediate computations is available by using $-notation. In particular, we can verify the observed counts, see the expected counts to see that all exceed 5, and look at the 'Pearson residuals'. The residuals are the signed square roots of the contributions $$C_{ij}$$ to the chi-squared statistic $$Q.$$ Specifically, we verify our computation of $$E_{11} = 353.67.$$ Also, recall that we computed $$C_{11} = 2.165,$$ and $$\sqrt{C_{11}} = 1.471.$$ The negative sign in the output signifies that $$X_{11} = E_{11} < 0.$$ Residuals with large expected values show where agreement between observed and expected count is most important. out$$obs [,1] [,2] [,3] A 326 319 355 B 348 350 302 C 387 310 303 out$$exp [,1] [,2] [,3] A 353.6667 326.3333 320 B 353.6667 326.3333 320 C 353.6667 326.3333 320 out$res
[,1]       [,2]       [,3]
A -1.4711595 -0.4059483  1.9565595
B -0.3013218  1.3101058 -1.0062306
C  1.7724814 -0.9041575 -0.9503289


Below is a graph of the density function of $$\mathsf{Chisq}(4).$$ The critical value is marked with a vertical dotted red line and the observed value of $$Q$$ is marked with a solid black line. 