Skip to main content
1 of 2
Zen
  • 25.1k
  • 4
  • 86
  • 126

Excuse me for changing your notation. You can do a simple Bayesian analysis of your data. Let the parameters $\Theta_1$ and $\Theta_2$ be the (unknown) click-through rates for each group. We will model $\Theta_1$ and $\Theta_2$ as independent $\textit{a priori}$, each of them with a $\mathrm{U}[0,1]$ distribution. You have a sample of click patterns of $n_1$ users from group 1 and $n_2$ users from group 2. Let $X_1$ and $X_2$ be the number of clicks observed in your sample for each group. We model the numbers of clicks on both groups as $$ X_1 \mid \Theta_1 = \theta_1 \sim \mathrm{Bin}(n_1,\theta_1) \qquad \qquad X_1 \mid \Theta_1 = \theta_1 \sim \mathrm{Bin}(n_2,\theta_2) \, . $$ After we know that $X_1=x_1$ and $X_2=x_2$, using Bayes's Theorem it is easy to show that $\textit{a posteriori}$ $\Theta_1$ and $\Theta_2$ are still independent, with posterior distributions $$ \Theta_1\mid X_1=x_1 \sim \mathrm{Beta}(x_1+1,n_1-x_1+1)\, , $$ $$ \Theta_2\mid X_2=x_2 \sim \mathrm{Beta}(x_2+1,n_1-x_2+1) \, . $$ From the joint posterior distribution of $\Theta_1$ and $\Theta_2$ you can compute any probability that you want. Suppose that you want know, for some $\epsilon>0$, what is the value of $$ P\{|\Theta_1-\Theta_2|>\epsilon\mid X_1=x_1,X_2=x_2\} \, . $$ You can get a Monte Carlo approximation of this probability sampling from the posterior. Here is an example in R.

N <- 10000 # number of Monte Carlo iterations
epsilon = 0.07
n1 <- 145
n2 <- 150
x1 <- 35 
x2 <- 50 
t1 <- rbeta(N, x1 + 1, n1 - x1 + 1)
t2 <- rbeta(N, x2 + 1, n2 - x2 + 1)
cat("Pr = ", sum(abs(t1 - t2) > epsilon) / N, "\n")

Which gives us the result:

Pr =  0.6564 
Zen
  • 25.1k
  • 4
  • 86
  • 126