Calculate the probability that the difference between two populations is >= d

Question

I'm a programmer, not a mathematician.

I have tried to use real math terms and notation here, but I may be misusing them.

I am doing split-testing, and I want a test to compare different click-through rates between two groups, A and B.

Existing tutorials and samples suggest a z-test or a t-test to reject the hypothesis that there was no change between A and B, but I have been trying to work through how to compare effect sizes rather than just "are the groups different?"

Let's say my two groups are sampled as ($a_k$ successes out of $a_n$ trials) and ($b_k$ successes out of $b_n$ trials). I am trying to determine differences between $A_p$ and $B_p$, the actual click-through probabilities of the two populations.

I have learned recently how to use the Beta distribution to model $A_p$ by setting $\alpha = a_k+1$ and $\beta = a_n-a_k+1$.

So, it seems to me that I can answer this question: "What is the probability that $B_p >= A_p + P_{delta}$?" by summing the probability of each candidate $A_p$ times the probability that $B_p$ is greater than $A_p+P_{delta}$:

$$\int_{p=0}^1 BetaPDF(p,a_k+1,a_n-a_k+1) (1 - BetaCDF(p+P_{delta}, b_k+1,b_n-b_k+1)) $$

Is this line of reasoning correct? It seems like a much more useful question to answer than what the normal t-test examples give you.

If this works, it seems like there must be a standard function/distribution/test for doing this?

I have written code to do this using a for loop with 1000 steps, but this is very slow when I want to graph over $P_{delta}$ (each datapoint in the graph means iterating the 1000 step for loop). I would appreciate having a better method.

Answer 1

Excuse me for changing your notation. You can do a simple Bayesian analysis of your data. Let the parameters $\Theta_1$ and $\Theta_2$ be the (unknown) click-through rates for each group. We will model $\Theta_1$ and $\Theta_2$ as independent $\textit{a priori}$, each of them with a $\mathrm{U}[0,1]$ distribution. You have a sample of click patterns of $n_1$ users from group 1 and $n_2$ users from group 2. Let $X_1$ and $X_2$ be the number of clicks observed in your sample for each group. We model the numbers of clicks on both groups as $$ X_1 \mid \Theta_1 = \theta_1 \sim \mathrm{Bin}(n_1,\theta_1) \qquad \qquad X_1 \mid \Theta_1 = \theta_1 \sim \mathrm{Bin}(n_2,\theta_2) \, . $$ After we know that $X_1=x_1$ and $X_2=x_2$, using Bayes's Theorem it is easy to show that $\textit{a posteriori}$ $\Theta_1$ and $\Theta_2$ are still independent, with posterior distributions $$ \Theta_1\mid X_1=x_1 \sim \mathrm{Beta}(x_1+1,n_1-x_1+1)\, , $$ $$ \Theta_2\mid X_2=x_2 \sim \mathrm{Beta}(x_2+1,n_2-x_2+1) \, . $$ From the joint posterior distribution of $\Theta_1$ and $\Theta_2$ you can compute any probability that you want. Suppose that you want know, for some $\epsilon>0$, what is the value of $$ P\{|\Theta_1-\Theta_2|>\epsilon\mid X_1=x_1,X_2=x_2\} \, . $$ You can get a Monte Carlo approximation of this probability sampling from the posterior. Here is an example in R.

N <- 10000 # number of Monte Carlo iterations
epsilon = 0.07
n1 <- 145
n2 <- 150
x1 <- 35 
x2 <- 50 
t1 <- rbeta(N, x1 + 1, n1 - x1 + 1)
t2 <- rbeta(N, x2 + 1, n2 - x2 + 1)
cat("Pr = ", sum(abs(t1 - t2) > epsilon) / N, "\n")

Which gives us the result:

Pr =  0.6564