Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am learning about AB testing using the G-test. In my example, I have a 2x2 contingency table.

>print(T)   
    response
AB    no yes Sum
  A   29   7  36
  B   23  16  39
  Sum 52  23  75

Event A is the red background of a website. Event B is the blue background of a website. I showed the website to the total of 75 people. Yes is like, and no is the opposite. After running the G-test I get

>likelihood.test(T)

Log likelihood ratio (G-test) test of independence without
    correction

data:  T
Log likelihood ratio statistic (G) = 4.1914, X-squared df = 1,
p-value = 0.04063

The p-value is pretty small (significant at the 5% level), so I reject the null that the samples A and B have the same performance. Now, I have two questions:

  1. How do I know what background color is better and has the higher performance?
  2. Why do I need the G test at all? I can just compare the percentage of likes for each background color. For red (A), 7/36=19.4%, and, for blue background (B), 16/39=41%. So, clearly B has the higher like percentage, hence, is better. So, why use G test at all?

Remark: I also use the Fisher exact test since one of the measured values, A_yes, is smaller than 10. The output is

> fisher.test(T)

    Fisher's Exact Test for Count Data

data:  T
p-value = 0.04948
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 0.9167091 9.6380957
sample estimates:
odds ratio 
  2.841057 

The p-value is nearly identical to the G-test's. So again why use the G-test or Fisher test at all when one can just compare the yes percentage for each event. Thanks.

share|improve this question
    
Which R packages' likelihood.test is this? The one in Deducer? –  Glen_b Jun 15 at 4:16

1 Answer 1

up vote 4 down vote accepted

Answering your two questions one at a time (though the answers are related):

  1. You only have data on whether they like it, not any information about 'performance'. But the hypothesis test(s) tell you whether or not the difference can be accounted for by chance, and you can tell the direction of the difference in the sample (the thing that would be leading to the rejection) by looking at the data.

  2. The problem is whether that apparent sample difference is just due to random variation. Imagine I toss two coins and one came up heads 80% of the time while the other came up heads 60% of the time. Is the first one more likely to come up heads in the long run? If the 80% and 60% was 4 heads and 3 heads in 5 tosses respectively, you could easily see that size of difference just by chance.

So it would be useful to have some idea of whether the difference is more than you could typically explain as nothing but chance. That's not necessarily best dealt with via hypothesis testing (you might look at something like a confidence interval for the difference, which also lets you see whether the size is big enough to regard as of practical importance.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.