Suppose you divide users into two groups by flipping a coin for each user. Now you measure the number of actions in each group (say, number of total clicks). How do you test if the two groups are different?
For the number of users in each group, I would use a chi-squared test between expected # users (total/2) and observed in each group.
What test can I use for the number of actions? The counts of actions are correlated, because the same user contributes multiple actions (clicks in our example). Moreover, it may be that some users contribute far more than others.
I could use a permutation test or a bootstrap, but that does not scale well with the number of users. (A chi-squared test that computes group sums on the database or Hive side is much easier than a bootstrap where maybe millions of user counts have to be transferred into R.)
Maybe I can use a chi-squared test anyway because the sum of # of actions should converge to normal (central limit theorem)?