Return to Answer

The test you probably want is Fisher's exact test. Unfortunately, given the likely very low click-through rate and small expected effect size, you will need enormous N to achieve the confidence interval you want. Lets say that the 'true' clickthrough rate of your best ad is .11, and your second best .1. Further, let's say you want the probability that you improperly reject the null hypothesis (that there is no difference between the two ads), to be less than .20. If this is so, you will need an N on the order of 10,000.

> library(statmod)   
> power.fisher.test(.1,.11,20000,20000,.05)
[1] 0.84

As a commenter suggested, you likely should not care about a ten percent difference in ad performance. For grosser differences, the necessary size of the samples decreases quickly.

> power.fisher.test(.1,.2,200,200,.05)
[1] 0.785

The test you probably want is Fisher's exact test. Unfortunately, given the likely very low click-through rate and small expected effect size, you will need enormous N to achieve the confidence interval you want. Lets say that the 'true' click-through rate of your best ad is .11, and your second best .1. Further, let's say you want the probability that you improperly fail to reject the null hypothesis (that there is no difference between the two ads), to be less than .20. If this is so, you will need an N on the order of 10,000.

> library(statmod)   
> power.fisher.test(.1,.11,20000,20000,.05)
[1] 0.84

As a commenter suggested, you likely should not care about a ten percent difference in ad performance. For grosser differences, the necessary size of the samples decreases quickly.

> power.fisher.test(.1,.2,200,200,.05)
[1] 0.785

The test you probably want is Fisher's exact test. Unfortunately, given the likely very low click-through rate and small expected effect size, you will need enormous N to achieve the confidence interval you want. Lets say that the 'true' clickthrough rate of your best ad is .11, and your second best .1. Further, let's say you want the probability that you improperly reject the null hypothesis (that there is no difference between the two ads), to be better than .80. If this is so, you will need an N on the order of 10,000.

> library(statmod)   
> power.fisher.test(.1,.11,20000,20000,.05)
[1] 0.84

As a commenter suggested, you likely should not care about a ten percent difference in ad performance. For grosser differences, the necessary size of the samples decreases quickly.

> power.fisher.test(.1,.2,200,200,.05)
[1] 0.785

The test you probably want is Fisher's exact test. Unfortunately, given the likely very low click-through rate and small expected effect size, you will need enormous N to achieve the confidence interval you want. Lets say that the 'true' clickthrough rate of your best ad is .11, and your second best .1. Further, let's say you want the probability that you improperly reject the null hypothesis (that there is no difference between the two ads), to be less than .20. If this is so, you will need an N on the order of 10,000.

> library(statmod)   
> power.fisher.test(.1,.11,20000,20000,.05)
[1] 0.84

As a commenter suggested, you likely should not care about a ten percent difference in ad performance. For grosser differences, the necessary size of the samples decreases quickly.

> power.fisher.test(.1,.2,200,200,.05)
[1] 0.785

The test you probably want is Fischer's exact test. Unfortunately, given the likely very low click-through rate and small expected effect size, you will need enormous N to achieve the confidence interval you want. Lets say that the 'true' clickthrough rate of your best ad is .11, and your second best .1. Further, let's say you want the probability that you improperly reject the null hypothesis (that there is no difference between the two ads), to be better than .80. If this is so, you will need an N on the order of 10,000.

> library(statmod)   
> power.fisher.test(.1,.11,20000,20000,.05)
[1] 0.84

As a commenter suggested, you likely should not care about a ten percent difference in ad performance. For grosser differences, the necessary size of the samples decreases quickly.

> power.fisher.test(.1,.2,200,200,.05)
[1] 0.785

The test you probably want is Fisher's exact test. Unfortunately, given the likely very low click-through rate and small expected effect size, you will need enormous N to achieve the confidence interval you want. Lets say that the 'true' clickthrough rate of your best ad is .11, and your second best .1. Further, let's say you want the probability that you improperly reject the null hypothesis (that there is no difference between the two ads), to be better than .80. If this is so, you will need an N on the order of 10,000.

> library(statmod)   
> power.fisher.test(.1,.11,20000,20000,.05)
[1] 0.84

As a commenter suggested, you likely should not care about a ten percent difference in ad performance. For grosser differences, the necessary size of the samples decreases quickly.

> power.fisher.test(.1,.2,200,200,.05)
[1] 0.785