BIC in Item Response Theory Models: Using log(N) vs log(N*I) as a weight

Question

In IRT software packages and in the literature it is common to calculate the BIC as

$$ \mathrm{BIC} = -2 \cdot \mathrm{logLik} + \log(N)\mathrm{Npars} $$

where $N$ is the number of rows in wide format (i.e. the number of persons) and $\mathrm{Npars}$ is the number of parameters. This is for example the way the R-packages ltm and TAM implement the BIC.

Typically IRT data is supplied in wide format, with one row per person and $I$ columns per item. If IRT data is reshaped to long format, it seems more natural to use $\log(N\cdot I)$ as a weight in the BIC criterion instead of $\log(N)$.

This is the perspective of generalized linear mixed models. In fact this is the output of the BIC method in lme4 if glmer is used to fit the Rasch model, as for example DeBoeck et al. (2011) illustrate in their JSS article on the estimation of IRT Models with glmer.

Clearly it could make a difference which weight is chosen, as the difference in BIC values is multiplied/divided by the factor $\log(N\cdot I)/\log(N)$. Is there an authorative source or some kind of argument in favour of one of the weights?

Answer 1

I think it's neither. The "textbook" information criteria formulation that you cite are derived for i.i.d. data, while you have a two-way array with weird cross-dependencies: you have the same questions, and you have the same students.

The issue is always there with mixed models. I am not going to try to reproduce their expressions, but Delattre et. al. (2014) paper (DOI:10.1214/14-EJS890) derives the relevant contributions from the observation-level (student-by-item) and cluster-level (student) data. They, however, seem to be oblivious to the prior work by the SAMSI group on Bayesian latent variable modeling, although frankly it did not do such a great job of documenting their results. The most important one is fully documented only in somebody's presentation. It was quoted by Jim Berger in the 2007 Wald lecture at the Joint Statistical Meetings. Finally, a rather lean book on random effect and latent variable model selection edited by David Dunson appear to have closely related results, but don't derive them in the form applicable to construction of the mixed-model BIC.