# Calculate the probability that the difference between two populations is >= d

I'm a programmer, not a mathematician.

I have tried to use real math terms and notation here, but I may be misusing them.

I am doing split-testing, and I want a test to compare different click-through rates between two groups, A and B.

Existing tutorials and samples suggest a z-test or a t-test to reject the hypothesis that there was no change between A and B, but I have been trying to work through how to compare effect sizes rather than just "are the groups different?"

Let's say my two groups are sampled as ($a_k$ successes out of $a_n$ trials) and ($b_k$ successes out of $b_n$ trials). I am trying to determine differences between $A_p$ and $B_p$, the actual click-through probabilities of the two populations.

I have learned recently how to use the Beta distribution to model $A_p$ by setting $\alpha = a_k+1$ and $\beta = a_n-a_k+1$.

So, it seems to me that I can answer this question: "What is the probability that $B_p >= A_p + P_{delta}$?" by summing the probability of each candidate $A_p$ times the probability that $B_p$ is greater than $A_p+P_{delta}$:

$$\int_{p=0}^1 BetaPDF(p,a_k+1,a_n-a_k+1) (1 - BetaCDF(p+P_{delta}, b_k+1,b_n-b_k+1))$$

Is this line of reasoning correct? It seems like a much more useful question to answer than what the normal t-test examples give you.

If this works, it seems like there must be a standard function/distribution/test for doing this?

I have written code to do this using a for loop with 1000 steps, but this is very slow when I want to graph over $P_{delta}$ (each datapoint in the graph means iterating the 1000 step for loop). I would appreciate having a better method.

Excuse me for changing your notation. You can do a simple Bayesian analysis of your data. Let the parameters $\Theta_1$ and $\Theta_2$ be the (unknown) click-through rates for each group. We will model $\Theta_1$ and $\Theta_2$ as independent $\textit{a priori}$, each of them with a $\mathrm{U}[0,1]$ distribution. You have a sample of click patterns of $n_1$ users from group 1 and $n_2$ users from group 2. Let $X_1$ and $X_2$ be the number of clicks observed in your sample for each group. We model the numbers of clicks on both groups as $$X_1 \mid \Theta_1 = \theta_1 \sim \mathrm{Bin}(n_1,\theta_1) \qquad \qquad X_1 \mid \Theta_1 = \theta_1 \sim \mathrm{Bin}(n_2,\theta_2) \, .$$ After we know that $X_1=x_1$ and $X_2=x_2$, using Bayes's Theorem it is easy to show that $\textit{a posteriori}$ $\Theta_1$ and $\Theta_2$ are still independent, with posterior distributions $$\Theta_1\mid X_1=x_1 \sim \mathrm{Beta}(x_1+1,n_1-x_1+1)\, ,$$ $$\Theta_2\mid X_2=x_2 \sim \mathrm{Beta}(x_2+1,n_2-x_2+1) \, .$$ From the joint posterior distribution of $\Theta_1$ and $\Theta_2$ you can compute any probability that you want. Suppose that you want know, for some $\epsilon>0$, what is the value of $$P\{|\Theta_1-\Theta_2|>\epsilon\mid X_1=x_1,X_2=x_2\} \, .$$ You can get a Monte Carlo approximation of this probability sampling from the posterior. Here is an example in R.

N <- 10000 # number of Monte Carlo iterations
epsilon = 0.07
n1 <- 145
n2 <- 150
x1 <- 35
x2 <- 50
t1 <- rbeta(N, x1 + 1, n1 - x1 + 1)
t2 <- rbeta(N, x2 + 1, n2 - x2 + 1)
cat("Pr = ", sum(abs(t1 - t2) > epsilon) / N, "\n")


Which gives us the result:

Pr =  0.6564

• +1. It may be useful to know that this difference of two betas (with integral parameters) has a piecewise polynomial PDF (and CDF): there are just two formulas, one for the negative values and another for the positive. A simple example: if $X\sim B(1,2)$ and $Y\sim B(2,3)$, then the CDF of $z=X-Y$ is $\frac{1}{5} (1+z)^4 \left(3-6 z+z^2\right)$ for $-1\le z\le 0$ and $\frac{1}{5} \left(3+6 z-5 z^2+2 z^5-z^6\right)$ for $0\le z\le 1$. For typical click-through data, though, approximating this as a Skellam distribution should be accurate and tractable. – whuber Mar 23 '13 at 16:55
• Thanks guys. There is a lot of stuff here that is over my head, but I will try to work through it. Does the piecewise representation of the CDF depend on the parameters to X and Y? When I was first working out this program, I plugged BetaRegularized[x,3,4] into Wolfram Alpha to see if it told me any polynomials for the Beta CDF. It does, but the polynomials are different when I use different values in place of the 3 and the 4. – Dusty Leary Mar 23 '13 at 18:22
• Good question, Dusty, and I'm sorry I wasn't clearer. Because a Beta has values between $0$ and $1$, a difference of Betas has values between $-1$ and $1$. There is a changeover in the distributional form of the difference precisely at $0$, no matter what the parameters of the two Betas might be. – whuber Mar 23 '13 at 23:50