I am running an experiment where I am testing the effects of three interventions (A,B,C) and measuring participants' performance via a multiple choice test with 5 questions. To perform hypothesis testing, I would like to use a 3-way ANOVA, however, I am faced with the challenge that I can't really assume my data is normal, since it can take on only 6 possible values (0-5 inclusive). I would like to use a Generalized linear model for my data instead, however, I'm not sure what kind of GLM I should use.

My first guess would be that this is a binomial distribution (how many successes out of n trials). However, my problem is that this would assume that all questions have equal difficulty/probability of success, which is not necessarily true. I have seen that Poisson and Inverse Binomial are often applied to count data, but my problem is that these distributions do not have an upper limit, while my data does. What distribution would be best for a GLM, or is there an alternative method that would be better?

  • 1
    $\begingroup$ 1. If the answers to the questions were independent, the number of successes would be a sum of independent Bernoulli's with different $p$, which is Poisson-binomial. $\,$ 2. Then there's also the question of whether the answers are in fact mutually independent, which might well not be true in general; answering one question may impact how you think about another for example. $\,$ One (perhaps rough) approximation for the number of successes that might perhaps be okay in some situations might be a quasi-binomial GLM. ... $\endgroup$
    – Glen_b
    Sep 4 at 2:46
  • $\begingroup$ ... $\,$ 4. You might model individual question difficulty and individual ability perhaps using something akin to a Bradley-Terry type model. Interactions between them become more difficult, you'd need to regularize. This possibility might be overkill if you're just interested in the total scores though, and still doesn't consider dependence. $\endgroup$
    – Glen_b
    Sep 4 at 2:49
  • $\begingroup$ If your question is whether interventions A, B, or C influences the choice of 0, 1, 2, 3, 4 or 5, then the response variable is categorical (and can take 6 values), and you need an ordinal regression. You are mentioning success or failure, I suppose this means the choice of 0, 1, 2, 3, 4 or 5 means some are wrong answers and some are right. If your question is then whether intervention A, B or C influences the probability of choosing the right answer between the 6 of them, the response variable is binomial (success/failure) and you can use a GLM for a binomial distribution : success/failure~in $\endgroup$
    – CaroZ
    Sep 4 at 11:07


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.