How to properly test for difference in number of user actions?

Question

Suppose you divide users into two groups by flipping a coin for each user. Now you measure the number of actions in each group (say, number of total clicks). How do you test if the two groups are different?

For the number of users in each group, I would use a chi-squared test between expected # users (total/2) and observed in each group.

What test can I use for the number of actions? The counts of actions are correlated, because the same user contributes multiple actions (clicks in our example). Moreover, it may be that some users contribute far more than others.

I could use a permutation test or a bootstrap, but that does not scale well with the number of users. (A chi-squared test that computes group sums on the database or Hive side is much easier than a bootstrap where maybe millions of user counts have to be transferred into R.)

Thoughts?

Maybe I can use a chi-squared test anyway because the sum of # of actions should converge to normal (central limit theorem)?

Answer 1

So you have two treatments (A and B) and randomize users into treatments groups$^\dagger$. The result is $n$ users (A) and $m$ users (B). For each user you observe a count variable $y_i$. If the counts are Poisson-distributed in each group (and independent between users) we can write the following model: $$ A \colon y_i \sim \text{Poisson}(\lambda), \quad i=1, \dotsc, n \\ B \colon y_i \sim \text{Poisson}(\mu), \quad i=n+1, \dotsc, n+m $$ but this is assuming the same Poisson mean for all users in same group, which might be too much to assume. But continuing under this assumption, we can get an exact test of $H_0 \colon \lambda=\mu$ in the usual way by using the binomial distribution of $\sum_{i\colon A} y_i$ conditional on $\sum_{i=1}^{n+m} y_i = Y$, which (under $H_0$) is $\text{Bin}(Y, \frac{n}{n+m})$. From that you can get p-values and confidence intervals (extending the argument to $\lambda \not = \mu$ case).

But the weakness of this argument is the assumption that the Poisson mean is constant in each group. If $\lambda_i, \mu_i$ has a gamma distribution in each group, then the count sums in each group has a negative-binomial distribution, and we could argue as above, but now the conditional distribution do not have a well-known, closed form. But it can be found (at least under $H_0$ of equality) using some special function (hypergeometric functions), I will come back to that. An alternative is a permutation test, but I would guess (will investigate) that the above proposed test based on negative binomial distributions could be a good approximation to a permutation test.

In this blog there is discussed a way of correcting the chi-squared test for overdispersion. That would probably be a simpler alternative.

$^\dagger$ In the comments is discussed the design via a randomized coin, which can lead to quite different $n$ and $m$. A better alternative could be a biased coin design.