Determining which AdWords have the highest amount of user click throughs

Question

Possible Duplicate

First, I am not a statistician (though I'd like be one) but I am trying to understand how different tests can be utilized to examine samples.

Let's say that I have five of Google AdWords ads and I'm trying to develop a test to show which has been more effective at getting users to click on that ad.

     January   February
 Ad A - 850       5000 

 Ad B - 900       5300

 Ad C - 880       5100

 Ad D - 880       5100  

 Ad E - 800       5000  

In terms of frequency, Ad B has the highest amount of user click throughs in both January and February. (Is it fine just to say that one has a higher freqency and is therefore the more effective ad without relying on any significance test)

However, for each month, I want to use a statistical test to determine which ad was more effective. Initially, I assumed a chi-square (pearsons) test would be appropriate, but with five seperate ads, I'm not sure if that's the appropriate plan of action.

Can anyone help me understand what action to take and why, or is it enough just to rely on the frequency distribution of the five ads.

Answer 1

I'm presuming the test you really want is whether B's apparent greater success can be regarded as statistically significant evidence that B generates more clicks in general; compared to a null hypothesis that all ads are effectively equal and B just had a lucky couple of months. Because if you just want to say "which ad had the most clicks?" the answer is trivial and is "B".

One standard way of testing this is through fitting a statistical model of some sort.

Your data show signs of increasing variance as their means increase, which suggests transforming it to a log scale. Then you can fit a model which assumes normal distribution of number of clicks, and use an ANOVA table and F test to test whether the "ad" factor is significant, or the variation seen between different ads may be due to chance.

Some code that does this in R is pasted below, and concludes that with a p value of 0.088 there is a reasonable chance that the data seen could have been generated from a random process where ad type does not influence the result.

> ad <- rep(LETTERS[1:5],2)
> mon <- rep(c("Jan", "Feb"),c(5,5))
> clicks <- c(850,900,880,880,800,5000,5300,5100,5100,5000)
> 
> model <- lm(log(clicks) ~ mon + ad)
...
> par(mfrow=c(2,2))
> plot(model) # check diagnostics
...
> summary(model)
...

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   8.5204     0.0171  497.23  9.8e-11 ***
monJan       -1.7784     0.0140 -127.10  2.3e-08 ***
adB           0.0577     0.0221    2.61    0.059 .  
adC           0.0272     0.0221    1.23    0.286    
adD           0.0272     0.0221    1.23    0.286    
adE          -0.0303     0.0221   -1.37    0.242    
---
... 

Residual standard error: 0.0221 on 4 degrees of freedom
Multiple R-squared:    1,       Adjusted R-squared: 0.999 
F-statistic: 3.23e+03 on 5 and 4 DF,  p-value: 2.67e-07 

> anova(model) # F test suggests p value of 0.088 for ad, so maybe difference is due to chance
Analysis of Variance Table

Response: log(clicks)
          Df Sum Sq Mean Sq  F value  Pr(>F)    
mon        1   7.91    7.91 16155.25 2.3e-08 ***
ad         4   0.01    0.00     4.49   0.088 .  
Residuals  4   0.00    0.00