# Can I use 3-way ANOVA for testing the difference between small proportions?

I'm looking at daily purchasing data over a period of several weeks. Specifically, we're interested in the % of users who make purchases in our app (it doesn't matter the type of purchase or amount spent for our purposes). On any given day, there might be ~10K active users and only about ~30 who actually make a purchase (it's a tough market!). There are three factors that we are interested, call them A, B, and C. Each factor has two levels.

A few rows of data look like:

#   NU      NP  %buy    A   B   C   date
1   12407   24  0.0019  0   0   2   2013-03-06
2   11774   37  0.0031  1   0   2   2013-03-05
3   12331   39  0.0032  0   0   2   2013-03-04
4   12679   41  0.0032  1   0   1   2013-03-03
5   12555   61  0.0049  0   1   2   2013-03-02
6   12594   50  0.0040  1   1   2   2013-03-01
7   12466   71  0.0057  0   1   2   2013-02-28
8   12089   61  0.0050  0   0   1   2013-02-27


Here, each row is a daily observation of one group of users with a set of A,B,C factors, NU is the number of total users in that group, NP is the number of users in the group that made a purchase, and %buy is just the % of users in the group who made a purchase. For convenience, let's assume that there are no important time-varying trends in any of the groups, which obviously could complicate the analysis even more (and over a short time period that we're looking at is probably an ok assumption anyway).

My first thought was to do 3-way ANOVA using the %buy as the observational outcome, but then I know I'm neglecting the importance of the sample sizes. Also, I'm thinking that the small proportions might mean a different test is necessary than run-of-the-mill ANOVA. So my question is really what is the proper test that I should be running here to look for significant differences between groups and within groups? Hopefully, I've given enough explanation to make this question clear.

Well, you can (I sure can't stop you!), but that doesn't mean you should.

Small proportions have variance almost proportional to their mean; ANOVA assumes constant variance.

They also tend to be fairly discrete.

There are equivalents of run-of-the-mill ANOVA for count data, using generalized linear models (GLMs). The basic understanding of it is broadly similar to doing ANOVA or any linear model, but it deals with (and allows inference on) more general distributions.

You could either model as Poisson with NU as exposure, or you could model as binomial. One nice thing about the Poisson is that even when the Poisson doesn't describe the data well (because the counts have substantively larger variance than the mean), you have the overdispersed Poisson; on the other hand if you want to model/think in terms of the proportions, then the binomial may be better.

So in R, for the Poisson, I might run a model like:

buy3.main <- glm(NP ~ A+B+C, family="poisson",offset=log(NU),data=purchasedata)

a main-effects only model, or

buy3.int <- glm(NP ~ A*B*C, family="poisson",offset=log(NU),data=purchasedata)

which is a model which includes all the interactions, or I could run various other models

One could then look at a comparison of models, via

anova(buy3.int,buy3.main)

One can also look at various contrasts or comparisons ... but what were F-tests become chi-square tests.

It's mostly a matter of learning about the differences for binomial and Poisson models; a lot of the intuition you likely already have should carry over.

• That's a great answer. Thanks so much for your insight. Makes a lot of sense to me. – thecity2 Mar 12 '13 at 16:08