Suppose I have a Likert scale type questionnaire that assesses the amount of social support a patient receives. The questionnaire is divided into four subscales that measure:

  • Emotional support
  • Instrumental support
  • Informational support
  • Appraisal support

At the end of the analysis, five scores are derived: one for each type of support, and an overall social support score.

I would then like to use IRT (Graded Response Model) to examine how well each question measures the latent types of support + overall social support.

All questions are positively correlated (although they correlate more positively within their respective subscales), and the questions fit into a bifactor model.

Should each subscale be analyzed separately? That is, should I conduct the IRT analysis separately for each of the four subscales? Or should the analysis be conducted at the overall level?

  • $\begingroup$ IRT is an analysis approach for non-validated instruments, but the MOSSS has been studied extensively and is well validated: that means that the sum score has a high degree of correlation with any latent structure built around the items, such as the type you'd get from GRMs. What is your goal with this analysis? $\endgroup$ – AdamO May 21 '19 at 14:40
  • 1
    $\begingroup$ A point of information: I believe that the GRM is irrelevant to the title. The issue is that you're looking at one overarching construct with 4 distinct but (strongly?) correlated dimensions. You would have the same issue if you were using the partial credit model or the non-Rasch version of the PCM. You would have the same issue if all the questions were binary and you were using the 1- or 2-PL model. $\endgroup$ – Weiwen Ng May 31 '19 at 12:27

Inference about the general factor (i.e. social support)

Your question is related to this recent question on unidimensionality for context. Your description of your goals could be clearer. However, it does sound like you want to estimate how well each item measures overall social support, while also considering that overall social support has the 4 correlated dimensions you enumerated. One option for that goal is the bifactor model. Reading list below.

Here is a short reading list on bifactor models. Any links are public access as far as I know.

Reise SP, Waller NG. Item Response Theory and Clinical Measurement. Annual Review of Clinical Psychology. 2009;5(1):27-48. This article discusses modern issues in using IRT in clinical settings, and it mentions the bifactor model (along with a bunch of other issues).

Reise, Moore and Haviland. Bifactor Models and Rotations: Exploring the Extent to which Multidimensional Data Yield Univocal Scale Scores The title is self-explanatory, but the authors do suggest that in many contexts, one of the normal unidimensional IRT models may be adequate to represent the overall construct.

Kirisci et al. Item Response Theory Analysis to Assess Dimensionality of Substance Use Disorder Abuse and Dependence Symptoms. These authors actually go and compare various IRT models on a panel of substance use questions. They fit a unidimensional model, comparing it with a bifactor model and a couple other multidimensional models.

Inference about the 4 subscales

It sounds like you also want to estimate the IRT parameters for the subscales. Here, I think that most people would just fit 4 IRT models separately and report the parameters. This seems entirely reasonable to me.

Going with the multidimensionality theme, it also seem to me like you could fit a generalized IRT model with correlated dimensions (e.g. Kirisci et al, in fig 1, their model C), and report the IRT parameters from there. You could also report the correlations of the 4 factors with each other. I think the factor correlations are substantively interesting, and should be reported. Remember, these correlations are set to zero in the bifactor model by definition, since (assuming I read the literature correctly) it's the general factor that's of real interest in that model.

That said, I am not sure if the correlated dimensions model will produce discrimination and difficulty parameter estimates consistent with the 4 separate IRT models. I think this should be the case, but I haven't done any reading on this specific issue.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.