How to analyse a “Check all that Apply” question

I have survey that asks respondents about the number of barriers they've experienced. The question lists different types of barriers, and respondents are asked to check each one they've experienced (so it's a "check all that apply" type question). They have 17 barriers to choose from. For each respondent, I've summed these to get the total number of barriers endorsed (which ranges from 0 - 17, M = 8.7, SD = 3.8). I would like to treat this as my DV in a mixed model. My question is, would it be best to analyze this type of outcome using an ordinal model or a poisson model? Or is this an empirical question I should explore during modeling?

• The rates seem a bit high for a Poisson to be a decent approximation. If that's your general approach, maybe a logistic regression, i.e. Binomial DV, would be better first cut. – conjugateprior Jul 6 '16 at 2:22
• When you say use a binomial, do you mean converting the DV to the proportion of barriers endorsed rather than using the number? – Michelle H Jul 7 '16 at 20:18
• Assuming 5 barriers were endorsed, the DV would be [5, 12], modeled by assuming that 5 was a draw from a Binomial with some endorsement probability - a function of covariates - and 17 'trials'. In R's lme4 package with a random effect for respondent and covariate covar that would be mod <- glmer(cbind(endorsed, 17-endorsed) ~ (1 | respondent) + covar, family=binomial) – conjugateprior Jul 7 '16 at 23:54

One approach is developed in papers by Wayne Desarbo, the only marketing scientist to be invited onto a Nobel Prize selection committee. He describes the method as appropriate for "pick any" choice data such as you have described. Here's the abstract to one of these papers:

This paper presents a new stochastic multidimensional scaling vector threshold model designed to analyze “pick any/n” choice data (e.g., consumers rendering buy/no buy decisions concerning a number of actual products). A maximum likelihood procedure is formulated to estimate a joint space of both individuals (represented as vectors) and stimuli (represented as points). The relevant psychometric literature concerning the spatial treatment of such binary choice data is reviewed. The nonlinear probit type model is described, as well as the conjugate gradient procedure used to estimate parameters.

The paper is titled A Stochastic Multidimensional Scaling Vector Threshold Model for the Spatial Representation of 'Pick Any/N' Data and is available here:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2785857

There are many more papers on various aspects of these algorithms.

• Not quite what I was looking for but interesting nonetheless. Thanks! – Michelle H Jul 7 '16 at 18:09

Each individual barrier is likely to be differently frequent, and each person to have a different propensity to endorse barriers, so your response is going to be a sum of binaries with different p(1). If your model captures the variation in p's you may be able to model the individual answers as Bernoulli in a mixed model, but even so, the sum will be Poisson binomial since the P's differ.

I'd be tempted to look at a Bayesian glmm for the individial answers - 17 per subject - estimated via some MCMC approach, and then sum them if need be.