I'm performing a web based a/b test where there is a control and one treatment. The results are not as simple as "converted" or "didn't convert." A user can "convert" anywhere between 0-10 times. I have all of the data from both the control and the treatment (how many times each user converted from both the control and the treatment). The sample sizes aren't the same size (about 1900 vs 2100) and the variances are different (7.12 vs 6.02). The mean of the treatment is about 11% higher than the control.
The goal of the experiment is to find out if the treatment can increase converts per user. The numbers are showing an 11% increase in conversions per user. To find out if the result is statistically significant I've been trying to use a Welch's t test. When I use the equation for a Welch's t test found on this wikipedia page I get the following results:
t-score: 2.26 degrees of freedom: 4025.82
On the wikipedia page it says I can use a
t-distribution to test the null hypothesis (my null hypothesis is that the means of the control and treatment are the same). But I'm not sure how to go about using a
t-distribution. I'm guessing it has something to do with using a t distribution table.
- Is using a Welch's t test a good approach for this situation?
- If not what other method would you suggest?
- With my results from the Welch's t test, how do I use a
t-distributionto determine if I have a significant result and with how much confidence?