How to properly test for difference in number of user actions?

Suppose you divide users into two groups by flipping a coin for each user. Now you measure the number of actions in each group (say, number of total clicks). How do you test if the two groups are different?

For the number of users in each group, I would use a chi-squared test between expected # users (total/2) and observed in each group.

What test can I use for the number of actions? The counts of actions are correlated, because the same user contributes multiple actions (clicks in our example). Moreover, it may be that some users contribute far more than others.

I could use a permutation test or a bootstrap, but that does not scale well with the number of users. (A chi-squared test that computes group sums on the database or Hive side is much easier than a bootstrap where maybe millions of user counts have to be transferred into R.)

Thoughts?

Maybe I can use a chi-squared test anyway because the sum of # of actions should converge to normal (central limit theorem)?

• Why would you not use a 'draw a card' model rather than a 'flip a coin' model for the assignment to treatment (which thereby guarantees equal assignment)? What's the precise hypothesis in the second test? Dec 31 '13 at 2:57
• Could you also explain what you mean by 'does not scale well with the number of users' for both the permutation test and the bootstrap? Dec 31 '13 at 3:13
• Imagine you want to run a test on m users, and you have a 'deck' of m=2n cards, n red and n black. Shuffle the cards. Every time you draw a red, you assign the user to treatment 1, and every time you draw a black assign the user to treatment 2. It's still randomization to treatment, but you don't have to worry that its unequal. Dec 31 '13 at 5:43
• @Glen: Maybe n is unknown. Dec 31 '13 at 15:31
• Databases (or Hive) are extremely efficient at computing counts per group. Dragging all those records to R makes the situation much worse, and in some cases impossible. Dec 31 '13 at 16:51

So you have two treatments (A and B) and randomize users into treatments groups$$^\dagger$$. The result is $$n$$ users (A) and $$m$$ users (B). For each user you observe a count variable $$y_i$$. If the counts are Poisson-distributed in each group (and independent between users) we can write the following model: $$A \colon y_i \sim \text{Poisson}(\lambda), \quad i=1, \dotsc, n \\ B \colon y_i \sim \text{Poisson}(\mu), \quad i=n+1, \dotsc, n+m$$ but this is assuming the same Poisson mean for all users in same group, which might be too much to assume. But continuing under this assumption, we can get an exact test of $$H_0 \colon \lambda=\mu$$ in the usual way by using the binomial distribution of $$\sum_{i\colon A} y_i$$ conditional on $$\sum_{i=1}^{n+m} y_i = Y$$, which (under $$H_0$$) is $$\text{Bin}(Y, \frac{n}{n+m})$$. From that you can get p-values and confidence intervals (extending the argument to $$\lambda \not = \mu$$ case).
But the weakness of this argument is the assumption that the Poisson mean is constant in each group. If $$\lambda_i, \mu_i$$ has a gamma distribution in each group, then the count sums in each group has a negative-binomial distribution, and we could argue as above, but now the conditional distribution do not have a well-known, closed form. But it can be found (at least under $$H_0$$ of equality) using some special function (hypergeometric functions), I will come back to that. An alternative is a permutation test, but I would guess (will investigate) that the above proposed test based on negative binomial distributions could be a good approximation to a permutation test.
$$^\dagger$$ In the comments is discussed the design via a randomized coin, which can lead to quite different $$n$$ and $$m$$. A better alternative could be a biased coin design.