3
$\begingroup$

I'm running an A/B test where I want to compare two different layout of a button and want to see if one layout lead to a higher Click-Through Rate with a certain statistical power. I want to use a significance level of 0.05.

Layout 1 has 25302 visualization and received 40 clicks. Layout 2 has 27924 visualization and received 50 clicks.

The distribution is binomial, considering the nature of the event I'm studying. Every visualization can in fact lead to a click or not.

I was trying to calculate it in R, using the pwr.2p2n.test() function included in the pwr library. The downside of this function is that it's asking the effect size as an input, which I don't know how to calculate.

$\endgroup$
4
  • 1
    $\begingroup$ Yes and no, I would like to know in the first place if the the pwr.2p2n.test() function the right solution. If that's the case I would like to know how to calculate the effect size parameter. $\endgroup$
    – Gianluca
    Jul 23, 2014 at 10:48
  • 1
    $\begingroup$ Power is not that relevant after the data are collected. I would stick to a confidence interval for the difference in two probabilities. $\endgroup$ Jul 23, 2014 at 13:03
  • 1
    $\begingroup$ The test is still running. I don't plan to stop it until the sample size is big enough to guarantee a statistical significance. My question is, how big have the samples be? or similarly, what's the statistical power at the moment? $\endgroup$
    – Gianluca
    Jul 23, 2014 at 15:57
  • 1
    $\begingroup$ You will do better in my opinion to state what you want to estimate, state the acceptable margin of error in estimating it, and then solve for $N$ that gives you that margin of error. The margin of error might be $\frac{1}{2}$ the width of the 0.95 confidence limit. The worst case margin of error is approximately obtained by setting $p=\frac{1}{2}$ and using $1.96 \sqrt(p(1-p)/n_{1} + p(1-p)/n_{2})$ where the two sample sizes are $n_{1}$ and $n_{2}$. You can set the ratio of the sample sizes to a constant $r$ and solve for one of the sample sizes, then the other. $\endgroup$ Jul 23, 2014 at 18:18

1 Answer 1

2
$\begingroup$

The effect size depends on how rare the Event is. Effect size specified as h parameter refers to cohen's d and is equal $h=2arcsin(\sqrt(p_1))-2arcsin(\sqrt(p_2))$, which means if the proportion in group A is 0.1 and You want to detect statistically significance difference in proportions within groups if the proportion in group B is at least bigger that 5% then You Have to substitute $p_1=0.1,p_2=0.15$ and then $h=0.15..$ and You Have to deliver such value of this parameter into that function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.