11
$\begingroup$

I am a software designer by trade and I am working on a project for a client, and I would like to make sure that my analysis is statistically sound.

Consider the following: We have n advertisements (n < 10), and we simply want to know which ad performs the best. Our ad server will randomly serve one of these ads. Success is if the a user clicks on the ad -- our server keeps track of that.

Given: Confidence Interval: 95%

Question: What is the estimated sample size? (How many total ads must we serve), Why? (remember i am a dummy)

Thanks

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Could you clarify what you means by "margin of error 5%"? $\endgroup$
    – onestop
    Commented Mar 10, 2011 at 20:12
  • $\begingroup$ @onestop -- good clarification -- I removed it from the question. I just took that variable from the following sample size calculator: raosoft.com/samplesize.html But I don't think its relavant in this question. Thanks! $\endgroup$
    – Jonathan
    Commented Mar 10, 2011 at 22:06
  • 2
    $\begingroup$ For many tests, you can calculate a sample size such that the test achieves a certain power given an assumed (fixed) effect size. In other words, you have to specifiy these things first: 1) what test do you want to use? 2) what power do you want that test to have? conditional on 3) an effect size that you deem interesting. 1) is something people here can probably help you with. 2) might be related to the 95% you indicated. 3) however, is something you have to provide beforehand: how different do the probabilities have to be to be considered interestingly different? $\endgroup$
    – caracal
    Commented Mar 11, 2011 at 11:16
  • $\begingroup$ So if i have to give more parameters here you go: 1. test to use -- no idea -- do you have suggestions? 2. power: even after looking at the wikipedia definition - I do not know how to intelligently answer that. 3. effect size: Lets say 10% better $\endgroup$
    – Jonathan
    Commented Mar 12, 2011 at 15:55

1 Answer 1

Reset to default
1
$\begingroup$

The test you probably want is Fisher's exact test. Unfortunately, given the likely very low click-through rate and small expected effect size, you will need enormous N to achieve the confidence interval you want. Lets say that the 'true' click-through rate of your best ad is .11, and your second best .1. Further, let's say you want the probability that you improperly fail to reject the null hypothesis (that there is no difference between the two ads), to be less than .20. If this is so, you will need an N on the order of 10,000.

> library(statmod)   
> power.fisher.test(.1,.11,20000,20000,.05)
[1] 0.84

As a commenter suggested, you likely should not care about a ten percent difference in ad performance. For grosser differences, the necessary size of the samples decreases quickly.

> power.fisher.test(.1,.2,200,200,.05)
[1] 0.785
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.