I'm a programmer, not a mathematician.

I have tried to use real math terms and notation here, but I may be misusing them.

I am doing split-testing, and I want a test to compare different click-through rates between two groups, A and B.

Existing tutorials and samples suggest a z-test or a t-test to reject the hypothesis that there was no change between A and B, but I have been trying to work through how to compare effect sizes rather than just "are the groups different?"

Let's say my two groups are sampled as ($a_k$ successes out of $a_n$ trials) and ($b_k$ successes out of $b_n$ trials). I am trying to determine differences between $A_p$ and $B_p$, the actual click-through probabilities of the two populations.

I have learned recently how to use the Beta distribution to model $A_p$ by setting $\alpha = a_k+1$ and $\beta = a_n-a_k+1$.

So, it seems to me that I can answer this question: "What is the probability that $B_p >= A_p + P_{delta}$?" by summing the probability of each candidate $A_p$ times the probability that $B_p$ is greater than $A_p+P_{delta}$:

$$\int_{p=0}^1 BetaPDF(p,a_k+1,a_n-a_k+1) (1 - BetaCDF(p+P_{delta}, b_k+1,b_n-b_k+1)) $$

Is this line of reasoning correct? It seems like a much more useful question to answer than what the normal t-test examples give you.

If this works, it seems like there must be a standard function/distribution/test for doing this?

I have written code to do this using a for loop with 1000 steps, but this is very slow when I want to graph over $P_{delta}$ (each datapoint in the graph means iterating the 1000 step for loop). I would appreciate having a better method.


Excuse me for changing your notation. You can do a simple Bayesian analysis of your data. Let the parameters $\Theta_1$ and $\Theta_2$ be the (unknown) click-through rates for each group. We will model $\Theta_1$ and $\Theta_2$ as independent $\textit{a priori}$, each of them with a $\mathrm{U}[0,1]$ distribution. You have a sample of click patterns of $n_1$ users from group 1 and $n_2$ users from group 2. Let $X_1$ and $X_2$ be the number of clicks observed in your sample for each group. We model the numbers of clicks on both groups as $$ X_1 \mid \Theta_1 = \theta_1 \sim \mathrm{Bin}(n_1,\theta_1) \qquad \qquad X_1 \mid \Theta_1 = \theta_1 \sim \mathrm{Bin}(n_2,\theta_2) \, . $$ After we know that $X_1=x_1$ and $X_2=x_2$, using Bayes's Theorem it is easy to show that $\textit{a posteriori}$ $\Theta_1$ and $\Theta_2$ are still independent, with posterior distributions $$ \Theta_1\mid X_1=x_1 \sim \mathrm{Beta}(x_1+1,n_1-x_1+1)\, , $$ $$ \Theta_2\mid X_2=x_2 \sim \mathrm{Beta}(x_2+1,n_2-x_2+1) \, . $$ From the joint posterior distribution of $\Theta_1$ and $\Theta_2$ you can compute any probability that you want. Suppose that you want know, for some $\epsilon>0$, what is the value of $$ P\{|\Theta_1-\Theta_2|>\epsilon\mid X_1=x_1,X_2=x_2\} \, . $$ You can get a Monte Carlo approximation of this probability sampling from the posterior. Here is an example in R.

N <- 10000 # number of Monte Carlo iterations
epsilon = 0.07
n1 <- 145
n2 <- 150
x1 <- 35 
x2 <- 50 
t1 <- rbeta(N, x1 + 1, n1 - x1 + 1)
t2 <- rbeta(N, x2 + 1, n2 - x2 + 1)
cat("Pr = ", sum(abs(t1 - t2) > epsilon) / N, "\n")

Which gives us the result:

Pr =  0.6564 
| cite | improve this answer | |
  • 1
    $\begingroup$ +1. It may be useful to know that this difference of two betas (with integral parameters) has a piecewise polynomial PDF (and CDF): there are just two formulas, one for the negative values and another for the positive. A simple example: if $X\sim B(1,2)$ and $Y\sim B(2,3)$, then the CDF of $z=X-Y$ is $\frac{1}{5} (1+z)^4 \left(3-6 z+z^2\right)$ for $-1\le z\le 0$ and $\frac{1}{5} \left(3+6 z-5 z^2+2 z^5-z^6\right)$ for $0\le z\le 1$. For typical click-through data, though, approximating this as a Skellam distribution should be accurate and tractable. $\endgroup$ – whuber Mar 23 '13 at 16:55
  • $\begingroup$ Thanks guys. There is a lot of stuff here that is over my head, but I will try to work through it. Does the piecewise representation of the CDF depend on the parameters to X and Y? When I was first working out this program, I plugged BetaRegularized[x,3,4] into Wolfram Alpha to see if it told me any polynomials for the Beta CDF. It does, but the polynomials are different when I use different values in place of the 3 and the 4. $\endgroup$ – Dusty Leary Mar 23 '13 at 18:22
  • $\begingroup$ Good question, Dusty, and I'm sorry I wasn't clearer. Because a Beta has values between $0$ and $1$, a difference of Betas has values between $-1$ and $1$. There is a changeover in the distributional form of the difference precisely at $0$, no matter what the parameters of the two Betas might be. $\endgroup$ – whuber Mar 23 '13 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.