I am struggling to choose the right model. In the data, 30 people answered 10 questions with either "Yes" or "No". The participants are seperated in two groups (Sizes: 13 and 17 per group). Also, there is a continious variable age. Some answers are missing. The question is, if there is a significant difference between the groups in their answers after controlling for age.

My first thought was to create a mixed-effect binary logistic regression model with random intercepts for participant and question:

glmer(answer ~ 1 + age + group + (1|ID) + (1|question), data=data, family=binomial)

I would then look at the p-value for the group variable. Would this work? It might be too complex for a bachelor thesis in a non-statistical field. I am not sure how I would report the result besides the p-value then and if I should add random slopes in the model as well.

My other idea is to simply run an ANCOVA with the overall rate of "Yes" answers (so someone answering "Yes" half of the time would get a y value of 0.5) as the dependant variable and age as the covariate. The target value's distribution would be heavily skewed though and not normally distributed since a lot of people answered "Yes". Is this problematic for doing ANCOVA?

Thanks, Fiona


1 Answer 1


This delves into the very complicated field of item response theory (IRT). IRT is the system of developing scoring algorithms from questionnaire data, addressing such issues as: is question A more discriminable than question B? or Is question C really necessary after asking questions A and B? or Does question D have the same sentiment as question A (reverse anchoring)? Based on your description of the problem, this may be too much into the weeds, but any approach you take needs to make some assumptions.

The apparent flexibility of the mixed logistic model can be quickly convoluted by the difficulty in interpreting the response. 1: there's no guarantee that the GLMM will actually converge - the model's performance can be poorly behaved in small samples with additional adjustments. 2: the estimated group effect - the log odds ratio of responding "Yes" in Group 1 to responding "Yes" in Group 0, is not intuitive and easily misinterpreted. 3: calculating the p-value from a mixed model is complicated and there is no omnibus. To consult the documentation from whatever your software is to provide the appropriate description is a statistician's job.

On the other hand, the simplicity of the ANCOVA can't be ignored - a simple sum-score is a very standard way to analyze multiple question questionnaires. If the particular question set administered is part of a standard instrument (like the CES-D for instance) you might already have scoring criteria for these questions. If not, you can still motivate the sum score as a pragmatic response variable. It should be obvious to the reader that a subject answering 5 yes questions has "more yes" than a subject answering 2 yes questions, regardless of whether the questions were equally difficult.

Regarding missing questionnaire data, hopefully all the questions are scanned for adequacy, such as asking the household income from a non-working parent - a common source of informative missingness. One you can use a number of approaches:

  • Standardize the sum score. For instance, a subject answering 9/9 questions affirmatively can be offset by 10/9 to reflect a 10 (i.e. 100%) "Yes" responses.
  • Use multiple imputation
  • Consider missing questions a negative.

$^*$ As this is thesis work, the best decision may well be whatever your advisor says.

  • $\begingroup$ Thanks! The questions are all very similiar so a sum score should make sense I think. I don't quite understand what you mean by Standardize the sum score" though. Does it mean that people with 9/9 will have a different score as someone with 10/10 then? $\endgroup$ Oct 27, 2022 at 9:02
  • $\begingroup$ @FionaGharamber no, standardization ensures that missing responses are mean imputed. This is inefficient and can be a little biased. As noted, if a subject answers 9 Yeses out of 9, then the best guess for the tenth missing response is also "Yes", and a 9/9 can be changed to 10/10 to handle the missing response. Similarly 0/9 would still be 0/10. $\endgroup$
    – AdamO
    Oct 27, 2022 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.