I ran an experiment on my website where I randomly assigned users to either the treatment or control group, and have two questions about how to correctly compute significance of the results.

Some context: The outcome variable is the total number of actions they took on my website. I am computing the mean number of actions taken by users in my control and treatment groups, respectively: $A_C = \frac{\sum_{i \in C} a_i }{|C|}$, $A_T = \frac{\sum_{i \in T} a_i}{|T|}$ (where $T$ and $C$ are my treatment and control groups and $a_i$ is the number of actions performed by user $i$). Is $A_T-A_C$ a reasonable outcome to measure? My concern is that since the $a_i$'s are distributed according to a heavy-tailed distribution, the mean is heavily skewed by a few power users. Usually I'd take the median, but the tails are so heavy that the median is very small.

Another thing I have measured is the KS statistic on the distributions of number of actions for users in the control and treatment groups.

  1. Is the difference in means or KS-statistic a more reasonable way to determine if the treatment had an effect?

  2. I am calculating bootstrapped p values by computing the same measures (difference in means and KS statistic) for many random splits of the user population, then counting how many random splits give rise to "more significant" values than my experimental control/treatment split gave. Is this correct?


There are two issues, though discussion of them are somewhat related:

i) What's a meaningful measure of treatment effect in the population?

ii) What's a good way to measure/test it in samples?

If you're especially interested in the difference in population means, then you should be looking to base your inference on that. On the other hand, often people really have a concept of just comparing the size "I want to know which one tends to be typically smaller". Indeed, a ratio of typical values may be just as meaningful to them as a difference, for example.

If the difference of population means is not especially more meaningful than say a ratio of means or a difference in medians, then you might consider a nonparametric test, such as a Wilcoxon-Mann-Whitney test (often called a Wilcoxon Rank-Sum test or a Mann-Whitney U test). This is good at pickup up differences and is not very badly affected by large outliers. In addition, it's equally adept at picking up that a ratio of typical values differs from 1 (this would be the same as looking at the difference of logs, and it will take the same value on the log scale as it would on the original scale). On the other hand it's also sensitive to other kinds of location measure than the mean, and can be conceived in terms of the probability that a random value from one population exceeds a random value from the other (and you can even write your hypotheses that way if you choose).

If the difference in population means is central, then you look at that, but as you note, in samples the sample means are sensitive to a few large values. This can certainly affect the usual t-test, but it's possible to use, for example, a permutation test or a bootstrap test (the two are quite similar).

A Kolmogorov-Smirnov test is sensitive to much more than a tendency for one group to be larger or smaller than the other. For example, if the treatment makes the observations more variable without making them larger, the KS test will still tend to reject the null - it will tell you they're different, because it's a test for equality of distributions. By having power against such alternatives, it has less power against alternatives that measure tendency to be larger or smaller.

The fact that you'd consider a Kolmogorov-Smirnov test makes me lean toward suggesting the Wilcoxon-Mann-Whitney as very likely a substantially better option.

If you haven't considered it, you might want to look into the possibility of matching or otherwise adjusting for use level. If you have some way of measuring/identifying the heavy users you could either match on it (and do a paired test), or you could use it as a covariate (or perhaps as an exposure measure) in a GLM. This will tend to give you better power to identify a treatment-difference.


This is an old thread but as lots of people are reading it...

From the description it sounds like you have a skewed distribution, not a heavy tailed distribution as those terms are commonly used. This matters as the remedies are different. For a skewed distribution, a common remedy is to transform the data to something like a normal distribution. Taking logs is one common such transformation. I have no way of knowing if logs would work here. Google "ladder of transformations".

Your "2" also sounds reasonable though it sounds like a permutation test rather than bootstrapping.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.