If I want to analyze a large sample size (N = 50.000) of continuous data ($ revenue) from an A/B test, what would then be the best way to check for normality?
Thanks in advance!
4
-
3$\begingroup$ Why do you want to check for normality? $\endgroup$– Stephan KolassaCommented Dec 25, 2019 at 10:27
-
$\begingroup$ to understand if I can use for example a T-test or if I need to use something like a mann-whitney u test $\endgroup$– JohnE230Commented Dec 25, 2019 at 10:30
-
2$\begingroup$ The t test does not require that the observations are normally distributed; it requires that the means are. By the Central Limit Theorem, they are so asymptotically even if the original observations have a quite non-normal distribution, under some quite weak assumptions. The Mann-Whitney test is nonparametric and makes no normality assumptions. You can probably use a t test out of the box, or possibly consider a permutation test for means (but see here). $\endgroup$– Stephan KolassaCommented Dec 25, 2019 at 10:41
-
1$\begingroup$ That the numerator in a t-test is close to normal is not of itself sufficient to make a t-statistic have a t-distribution. To get that, you really do need the observations to be normal. Beyond that, in large samples you have a justification for an approximate z-test (CLT + Slutsky), not a t-test. Is there some argument that shows that t will be better than z in general in that situation? $\endgroup$– Glen_bCommented Dec 25, 2019 at 11:42
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The comments are all worthwhile and you may not need to test for normality at all.
But if you are comparing revenue from two groups (A and B) then I would be amazed if it were normally distributed. In my experience, money variables are almost never normally distributed - they tend to have long tails to the right and to have outliers.
I would also wonder if comparing the mean revenue (which is what a t-test does) is really what you want (or, if it is wanted, if you might also want something else).
Testing normality for large samples has its own problems. Tests (like Shapiro Wilk and so on) are likely to give significant results (i.e. not normal) for trivial deviations from normality. Graphical methods (such as quantile normal plots) are better, but require some experience and judgment to read.
Given all this, why not go to quantile regression? It doesn't make assumptions about the residuals and gives you (at least optionally) more information?
answered Dec 25, 2019 at 13:16