Determining significance and variance in A B C D testing

Question

Scenario:

An email campaign where four different email designs (treatments) are sent to four different populations of equal size in an attempt to find which performs better.

The results returned are:

Treatment     Population        Clicks 
A             120 (Group 1)     45 
B             120 (Group 2)     35
C             120 (Group 3)     68
D             120 (Group 4)     52

Goals:

1. To figure out if any treatments perform (statistically) significantly better then others.

2. To find the amount of random variance expected in the test.

Problem:

While I can easily see treatment C appears to perform better, I'm aware that there will be a number of clicks simply due to 'chance' (or factors not related to the treatment) in each test.

Given the small population, my gut tells me if that if I could run the test again it's quite possible that the results could indicate another winner.

Essentially I'm trying to determine how much variability I should expect and factor into the results to determine if the winner is actually a winner or just ahead due to chance.

Approaches:

I have only basic stats understand and have tried the following ideas:

• looked at approaching this through standard deviation, confidence limits and ANOVA, however given that there is only one data point per treatment these don't seem like the correct tools.
• thought about conducting an A|A|A|A test next time, where the same treatment is sent to different groups and see what the variance is, then use that as fuzz factor

Any suggestions on the correct tool to apply to achieve what I'm after would be hugely appreciated!

Answer 1

I eventually figured out a 'Chi Squared Test for Independence' was what I was looking for.

This allowed me to test and see if the results were likely to be due to chance or not.