# Analyze results from a/b test using welch's t test

I'm performing a web based a/b test where there is a control and one treatment. The results are not as simple as "converted" or "didn't convert." A user can "convert" anywhere between 0-10 times. I have all of the data from both the control and the treatment (how many times each user converted from both the control and the treatment). The sample sizes aren't the same size (about 1900 vs 2100) and the variances are different (7.12 vs 6.02). The mean of the treatment is about 11% higher than the control.

The goal of the experiment is to find out if the treatment can increase converts per user. The numbers are showing an 11% increase in conversions per user. To find out if the result is statistically significant I've been trying to use a Welch's t test. When I use the equation for a Welch's t test found on this wikipedia page I get the following results:

t-score: 2.26
degrees of freedom: 4025.82


On the wikipedia page it says I can use a t-distribution to test the null hypothesis (my null hypothesis is that the means of the control and treatment are the same). But I'm not sure how to go about using a t-distribution. I'm guessing it has something to do with using a t distribution table.

Questions:

• Is using a Welch's t test a good approach for this situation?
• If not what other method would you suggest?
• With my results from the Welch's t test, how do I use a t-distribution to determine if I have a significant result and with how much confidence?
• May I ask what led you to choose a Welch t-test instead of a classical t-test (read, how would you justify your claim that "the variances are different" from a statistical perspective? Did you perform any test--the ratio of the two variances is about 1.2?)? Also, it seems you outcome is a discrete bounded variable (0-10), can you confirm this? – chl Dec 13 '12 at 22:47
• @chl I am not completely set on using a Welch's t-test. I'm trying to find out what the best test for this situation would be. On this wikipedia page it says if you have unequal sample sizes and unequal variance that you should use a Welch's t-test. Maybe I don't understand what constitutes unequal variance, all I've done is calculate the variance of the control and the treatment and they are "different" (7.12 vs 6.02). But maybe I'm going about that the wrong way. – nates Dec 14 '12 at 16:24
• @chl to answer your question about the outcome variable, each user is bound to only "convert" 0-10 times. But each user is independent from all others. Does that answer your question? – nates Dec 14 '12 at 16:26
• @nates, maybe I'm answering it way too late, but for the record and to help people looking for this answer in the future: to determine unequal variance you use a heteroskedasticity test to see if the variances are significantly unequal. For that, a Levene's test is typically used. However, there are several discussions on whether using Levene's test prior to a Welch's t or a Student's t is really a good idea. For reference look at daniellakens.blogspot.com/2015/01/… – Bruno Nov 3 '18 at 5:18