Possible Duplicate

First, I am not a statistician (though I'd like be one) but I am trying to understand how different tests can be utilized to examine samples.

Let's say that I have five of Google AdWords ads and I'm trying to develop a test to show which has been more effective at getting users to click on that ad.

     January   February
 Ad A - 850       5000 

 Ad B - 900       5300

 Ad C - 880       5100

 Ad D - 880       5100  

 Ad E - 800       5000  

In terms of frequency, Ad B has the highest amount of user click throughs in both January and February. (Is it fine just to say that one has a higher freqency and is therefore the more effective ad without relying on any significance test)

However, for each month, I want to use a statistical test to determine which ad was more effective. Initially, I assumed a chi-square (pearsons) test would be appropriate, but with five seperate ads, I'm not sure if that's the appropriate plan of action.

Can anyone help me understand what action to take and why, or is it enough just to rely on the frequency distribution of the five ads.

  • $\begingroup$ Thanks for the link to a possible duplicate. The questions are related but they are not duplicates: here, you are asking for how to analyze the data, not for a sample size. $\endgroup$ – whuber May 7 '11 at 15:33
  • $\begingroup$ To calculate chi-square paired tests you need impressions and clicks for each ad. $\endgroup$ – jrosell Oct 9 '14 at 23:29

I'm presuming the test you really want is whether B's apparent greater success can be regarded as statistically significant evidence that B generates more clicks in general; compared to a null hypothesis that all ads are effectively equal and B just had a lucky couple of months. Because if you just want to say "which ad had the most clicks?" the answer is trivial and is "B".

One standard way of testing this is through fitting a statistical model of some sort.

Your data show signs of increasing variance as their means increase, which suggests transforming it to a log scale. Then you can fit a model which assumes normal distribution of number of clicks, and use an ANOVA table and F test to test whether the "ad" factor is significant, or the variation seen between different ads may be due to chance.

Some code that does this in R is pasted below, and concludes that with a p value of 0.088 there is a reasonable chance that the data seen could have been generated from a random process where ad type does not influence the result.

> ad <- rep(LETTERS[1:5],2)
> mon <- rep(c("Jan", "Feb"),c(5,5))
> clicks <- c(850,900,880,880,800,5000,5300,5100,5100,5000)
> model <- lm(log(clicks) ~ mon + ad)
> par(mfrow=c(2,2))
> plot(model) # check diagnostics
> summary(model)

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   8.5204     0.0171  497.23  9.8e-11 ***
monJan       -1.7784     0.0140 -127.10  2.3e-08 ***
adB           0.0577     0.0221    2.61    0.059 .  
adC           0.0272     0.0221    1.23    0.286    
adD           0.0272     0.0221    1.23    0.286    
adE          -0.0303     0.0221   -1.37    0.242    

Residual standard error: 0.0221 on 4 degrees of freedom
Multiple R-squared:    1,       Adjusted R-squared: 0.999 
F-statistic: 3.23e+03 on 5 and 4 DF,  p-value: 2.67e-07 

> anova(model) # F test suggests p value of 0.088 for ad, so maybe difference is due to chance
Analysis of Variance Table

Response: log(clicks)
          Df Sum Sq Mean Sq  F value  Pr(>F)    
mon        1   7.91    7.91 16155.25 2.3e-08 ***
ad         4   0.01    0.00     4.49   0.088 .  
Residuals  4   0.00    0.00        
  • $\begingroup$ Your answer is ok if we are talking clics. Chi-square is more appropiate for CTR (clicks/impressions). $\endgroup$ – jrosell Oct 9 '14 at 23:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.