Analyze results from a/b test using welch's t test I'm performing a web based a/b test where there is a control and one treatment. The results are not as simple as "converted" or "didn't convert." A user can "convert" anywhere between 0-10 times. I have all of the data from both the control and the treatment (how many times each user converted from both the control and the treatment). The sample sizes aren't the same size (about 1900 vs 2100) and the variances are different (7.12 vs 6.02). The mean of the treatment is about 11% higher than the control.
The goal of the experiment is to find out if the treatment can increase converts per user. The numbers are showing an 11% increase in conversions per user. To find out if the result is statistically significant I've been trying to use a Welch's t test. When I use the equation for a Welch's t test found on this wikipedia page I get the following results:
t-score: 2.26
degrees of freedom: 4025.82

On the wikipedia page it says I can use a t-distribution to test the null hypothesis (my null hypothesis is that the means of the control and treatment are the same). But I'm not sure how to go about using a t-distribution. I'm guessing it has something to do with using a t distribution table.
Questions:


*

*Is using a Welch's t test a good approach for this situation? 

*

*If not what other method would you suggest?


*With my results from the Welch's t test, how do I use a t-distribution to determine if I have a significant result and with how much confidence?

 A: *

*Technically the Welch test assumes that your data is numeric, whereas your data is constrained to take integer values between 0 and 10 (if the 10 conversions are independent then the data for each of your observations is binomial).  A better test is The Mann-Whitney U Test.  For the type of data that you have you need a version that deals with ties in the data (i.e., situations where respondents are able to have the same value).  Do a search for "IBM SPSS Mann-Whitney Test" and you should find the algorithm (unfortunately, IBM do not permit links to their web pages so I have not provided the URL).  Having said all of that, you will be highly likely to get the same answer, as your large sample size means that the assumptions of the Welch test aren't likely to be problematic; I would only bother with the Mann-Whitney if you are likely to need to have your work reviewed.

*There are lots of t-tables on the web and in the back of stats books.  However, in practice most people look them up in software.  For example, in Excel enter "=T.DIST.2T(2.26,4025.82)" and you will compute your p-value as 0.023874.  If expressed as confidence, you would say 97.6%, but the whole notion of "confidence" in this context is pretty dodgy in my view and I think reporting the p-value is less ambiguous.

