# How to calculate practical significance on simple A/B email marketing test?

This might be a newbie question, but I'm not great in statistics:

I have this test for an email marketing. It was a simple test to understand whether people would purchase more through an email marketing campaign with and without an image banner. So the dependent variable is clicked emails and the independent variable is the banner.

List size (total population): 20,210 Total number of people who were picked to receive the email for the test: 10,882 (53% of the sample size)

Number of people which were sent email A: 5,441 Number of people who opened email A: 992 Number of people who clicked on email A:58

Number of people which were sent email B: 5,441 Number of people who opened email B: 991 Number of people who clicked on email B: 39

The click rate is 1.1% on A, and 0.7% on B, but how do I know if this has practical importance? can I conclude that this test tells me that is more effective to use banners? If not how do I fix this experiment?

For a simple significance test (absent of statistical sampling), use the Chi-Square test. Instead of using who received the email, I'm going to use who actually opened the email.

Step 1: figure out Expected Value of each:

A = 58/(992+991) * (58+39) = 78.2
B = 39/(992+991) * (58+39) = 76.8


Step 2: Chi-sqr

= ((78.2-58)^2) + ((76.8-39)^2) = 3.6


Step 3: compare value in Step 2 to Chi-sqr @ 95% Confidence

Chi_sqr @ 95% value = 3.84

Since 3.6 < 3.84, the result is cannot conclude the A performed better than B, or not significant.

• I guess I need to repeat the experiment with a larger sample size? Commented Dec 22, 2015 at 18:55
• yes, you can play with the #s in my solution to determine sample size needed when "Step 2" > 3.84 Commented Dec 22, 2015 at 21:09

Try the wikipedia page for A/B testing to give you an idea of which test to use for which metric of interest. For 'click-thru-rate' use an online calculator for the ChiSquare test here's one

         |  Clicked   Didn't