How to calculate practical significance on simple A/B email marketing test?

Question

This might be a newbie question, but I'm not great in statistics:

I have this test for an email marketing. It was a simple test to understand whether people would purchase more through an email marketing campaign with and without an image banner. So the dependent variable is clicked emails and the independent variable is the banner.

List size (total population): 20,210 Total number of people who were picked to receive the email for the test: 10,882 (53% of the sample size)

Number of people which were sent email A: 5,441 Number of people who opened email A: 992 Number of people who clicked on email A:58

Number of people which were sent email B: 5,441 Number of people who opened email B: 991 Number of people who clicked on email B: 39

The click rate is 1.1% on A, and 0.7% on B, but how do I know if this has practical importance? can I conclude that this test tells me that is more effective to use banners? If not how do I fix this experiment?

Answer 1

For a simple significance test (absent of statistical sampling), use the Chi-Square test. Instead of using who received the email, I'm going to use who actually opened the email.

Step 1: figure out Expected Value of each:

A = 58/(992+991) * (58+39) = 78.2
B = 39/(992+991) * (58+39) = 76.8

Step 2: Chi-sqr

= ((78.2-58)^2) + ((76.8-39)^2) = 3.6

Step 3: compare value in Step 2 to Chi-sqr @ 95% Confidence

Chi_sqr @ 95% value = 3.84

Since 3.6 < 3.84, the result is cannot conclude the A performed better than B, or not significant.  

Answer 2

Try the wikipedia page for A/B testing to give you an idea of which test to use for which metric of interest. For 'click-thru-rate' use an online calculator for the ChiSquare test here's one

Setup your table like this:

         |  Clicked   Didn't
----------------------------
Group 1  |      39     952
Group 2  |      58     933

The chi-square statistic is 3.9132. The p-value is .047909. This result is significant at p < .05.

This shows statistical significance, not practical importance.

Statistical significance says: differences are probably not from random sampling. But practical importance would say: yes, include the banner because having 58 people click-through is better than 39.