I have a study where each of the 43 items is a question to which the response may be A or B and I want to infer whether A has been chosen significantly more than B. There are 20 participants and 43 items, I have thus 20*43= 860 rows in my datafile. I know I can run a binomial test/chi square, but my problem is that these tests don't take into account the unmeasured correlation between responses due to participants and questions.


You can use a mixed effects model, a logistic regression with random intercepts for question and for participant. If you have data in long format, something like

participant  question answer
1            1        A
1            2        B
n            1        A
n            2        A

the logistic regression will have a linear predictor $$ \eta_{ij} = \mu + \alpha_i + \beta_j $$ where $\alpha_i$ is a random intercept for participant $i$ and $\beta_j$ a random intercept for question $j$. Then the model is completed by stipulating that $$ \alpha_i \stackrel{\text{iid}}{\sim} \mathcal{N}(0,\sigma^2_\alpha) \\ \beta_j \stackrel{\text{iid}}{\sim} \mathcal{N}(0,\sigma^2_\beta) $$

If you have some other covariates you can build from here.

In R something like

df <- <your data frame>
mod0 <- glmer( answer ~ (1 | participant) + (1 | question), 
               data=df, family=binomial )

This is similar to the setting in item response theory, so look through . See also the tag , since such rating systems are based on similar models. An interesting applications is the chess puzzles on lichess where both solver and quizz will get its own random parameter (but they will be updated according to how the puzzle is solved).

  • $\begingroup$ Not very familiar to R (python user...) and had a curiosity. "(1 | participant)" is kind like of a one hot encoding where you add n_participants dummy features ? Actually I have also an other question. $\alpha_i$ is the "trait" discussed here en.wikipedia.org/wiki/Item_response_theory ? Can we interpret it as the intelligence of the student ? $\endgroup$ – Thomas Feb 27 at 15:29
  • $\begingroup$ @Thomas: That notation means a random intercept for each participant and a typical implementation migfht represent it like you say (bur in R package lme4 it uses sparse matrices, which is part of the reason for its speed). But the estimation is still different, since it is taken as a random not fixed effect. If you can interpret $\alpha_i$ is intelligence I am sure will be debated, and depend on context and subject matter knowledge not statistics. It might certainly be influenced also by experience, knowledge, industriousness, seriousness of study, ... $\endgroup$ – kjetil b halvorsen Feb 27 at 15:36
  • $\begingroup$ Thank you for your answer. Do you have a reference/link that explains the difference between a "random intercept" and just a "fixed, but student dependent, intercept" ? Admittedly, I am not appreciating the difference right now and I am curious to know more ... $\endgroup$ – Thomas Feb 27 at 15:41
  • $\begingroup$ @Thomas: I updated my answer to include more details. Have a look at stats.stackexchange.com/questions/26230/… and stats.stackexchange.com/questions/4700/… $\endgroup$ – kjetil b halvorsen Feb 27 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.