5
$\begingroup$

I'm performing a web based a/b test where there is a control and one treatment. The results are not as simple as "converted" or "didn't convert." A user can "convert" anywhere between 0-10 times. I have all of the data from both the control and the treatment (how many times each user converted from both the control and the treatment). The sample sizes aren't the same size (about 1900 vs 2100) and the variances are different (7.12 vs 6.02). The mean of the treatment is about 11% higher than the control.

The goal of the experiment is to find out if the treatment can increase converts per user. The numbers are showing an 11% increase in conversions per user. To find out if the result is statistically significant I've been trying to use a Welch's t test. When I use the equation for a Welch's t test found on this wikipedia page I get the following results:

t-score: 2.26
degrees of freedom: 4025.82

On the wikipedia page it says I can use a t-distribution to test the null hypothesis (my null hypothesis is that the means of the control and treatment are the same). But I'm not sure how to go about using a t-distribution. I'm guessing it has something to do with using a t distribution table.

Questions:

  • Is using a Welch's t test a good approach for this situation?
    • If not what other method would you suggest?
  • With my results from the Welch's t test, how do I use a t-distribution to determine if I have a significant result and with how much confidence?
$\endgroup$
  • $\begingroup$ May I ask what led you to choose a Welch t-test instead of a classical t-test (read, how would you justify your claim that "the variances are different" from a statistical perspective? Did you perform any test--the ratio of the two variances is about 1.2?)? Also, it seems you outcome is a discrete bounded variable (0-10), can you confirm this? $\endgroup$ – chl Dec 13 '12 at 22:47
  • $\begingroup$ @chl I am not completely set on using a Welch's t-test. I'm trying to find out what the best test for this situation would be. On this wikipedia page it says if you have unequal sample sizes and unequal variance that you should use a Welch's t-test. Maybe I don't understand what constitutes unequal variance, all I've done is calculate the variance of the control and the treatment and they are "different" (7.12 vs 6.02). But maybe I'm going about that the wrong way. $\endgroup$ – nates Dec 14 '12 at 16:24
  • $\begingroup$ @chl to answer your question about the outcome variable, each user is bound to only "convert" 0-10 times. But each user is independent from all others. Does that answer your question? $\endgroup$ – nates Dec 14 '12 at 16:26
  • $\begingroup$ @nates, maybe I'm answering it way too late, but for the record and to help people looking for this answer in the future: to determine unequal variance you use a heteroskedasticity test to see if the variances are significantly unequal. For that, a Levene's test is typically used. However, there are several discussions on whether using Levene's test prior to a Welch's t or a Student's t is really a good idea. For reference look at daniellakens.blogspot.com/2015/01/… $\endgroup$ – Bruno Nov 3 '18 at 5:18
2
$\begingroup$
  1. Technically the Welch test assumes that your data is numeric, whereas your data is constrained to take integer values between 0 and 10 (if the 10 conversions are independent then the data for each of your observations is binomial). A better test is The Mann-Whitney U Test. For the type of data that you have you need a version that deals with ties in the data (i.e., situations where respondents are able to have the same value). Do a search for "IBM SPSS Mann-Whitney Test" and you should find the algorithm (unfortunately, IBM do not permit links to their web pages so I have not provided the URL). Having said all of that, you will be highly likely to get the same answer, as your large sample size means that the assumptions of the Welch test aren't likely to be problematic; I would only bother with the Mann-Whitney if you are likely to need to have your work reviewed.
  2. There are lots of t-tables on the web and in the back of stats books. However, in practice most people look them up in software. For example, in Excel enter "=T.DIST.2T(2.26,4025.82)" and you will compute your p-value as 0.023874. If expressed as confidence, you would say 97.6%, but the whole notion of "confidence" in this context is pretty dodgy in my view and I think reporting the p-value is less ambiguous.
$\endgroup$
  • $\begingroup$ Thank you for your response. The 10 conversions are not independent (it sounds like you made the correct assumption but I just wanted to clarify). I read about the Mann-Whitney U Test and one of the requirements is that the data is ordinal. I guess I could set it up so that there are a total of 11 ranks (0-10 conversions) but then there will be hundreds of ties for each rank. This seems like the wrong way to go to me, but I'm not a statistician. Is it a problem that each rank would have many "ties?" $\endgroup$ – nates Dec 19 '12 at 16:24
  • $\begingroup$ So long as you use a formula that adjusts for the number of ties, the actual number is immaterial. $\endgroup$ – Tim Feb 6 '13 at 6:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.