How do I test for significant change when looking at a metric such as "average products added to cart in per session"? I will try to ask this question via an example. Let us assume we have two groups. Group A & Group B.




Group
Total Products added to Cart
Total Sessions
Avg. products added to cart per session




A
25,000
21,000
1.1905


B
27,500
21,955
1.2526




What I want to understand is, is the difference in avg. products added to cart per session between Group A and Group B, statistically significant?
My initial thoughts was to use unpaired 2-sample t-test. But I am questioning this more and more...mainly because I am getting confused as to whether it is actually appropriate to calculate the std. deviation of the 'average prod. added to cart per session'? Usually in t-tests you calculate the std. deviation of a value such as weight, test scores and not average weight or average score.
Could any one please advise and help me?
 A: This is like, THE QUESTION, in AB testing.  There are a bunch of ways to answer this, each of which have their pros and cons.  I'll give my thoughts here for posterity.
The t-test
Perhaps the most common approach, even among enterprise software which offer experimentation platforms, is to use the t-test.  Yadayadayada Central Limit Theorem, Yadayadayada asymptotic normality, yadayadayada t-test.  The rationale for using this approach can be found here so I won't belabour the point.
Here is the thing.  In internet experiments, it is very common that the distribution from which your data come from is long tailed.  This means that the normality assumption underlying the t-test can fail and can fail in such a way that the validity of the inference comes into question.  Remember, the CLT tells us that the sampling distribution of the mean is normal, but it doesn't tell us when that happens.
Poisson Regresson and Variants of a GLM
You're counting stuff in the end, so it makes sense that a model like
$$ \log(E(y)) = \beta_{0} + \beta_1 \mbox{Txt} + \log(\mbox{Total Sessions}) $$
could be applied using Poisson regression or Negative binomial regression.  Here, $\beta_1$ is the effect of the variant and the quantity in the log is called an "offset".  Its meant to account for the fact that more products will be added to the cart if you have more sessions.
The benefit of this approach is that you can adjust for customer level variables (like age, past spend, etc) which can further reduce the variance in the outcome, increasing the power of your experiment.  You'd probably need to compute a marginal effect to get an estimate of the average treatment effect however, but that is very easy with R packages.
Winsorization and Adaptive Trimmed Means
Most people will probably reach for winsorization, but I'm dubious of that approach.  I'm dubious because most approaches I have seen just winsorize the data at the 1% and 99% percentiles (or similar) and then go on computing the t test or similar.  However, I'm not convinced that is correct, because the resulting estimate of the standard error does not account for the winsorizing process and is likely an underestimation of the sampling variance of the test statistic.  Now, there may be a solution around this.  I just haven't seen it.
What I have seen is an approach specified in this thread and in this paper.  The approach is similar to winsorization, but instead of clipping the data at percentiles we simply discard the data above and below these quantiles.  The best part is that we can analytically calculate the variance of this procedure, and the simulations shown in that twitter thread I've linked looked very promising.
Which to Use
I don't know.  I'm quite fond of the second approach, but only because we reduce the variance quite a lot by controlling for pre-exposure average products added to the cart per session.  This is a process called "CUPED" in recent papers and should remove the need to adjust for other customer level factors.  All you would need is the treatment effect, the effect of pre-exposure items per session, the intercept, and the offset, which sounds enticing but is subject to assumptions on the likelihood.
Its going to depend on your domain expertise and your data.  In any case, you will need to validate the assumptions you make.
