# What are typical approaches to express the reliability of a categorical scale?

I have categorical scales consisting of items with each three answer options: 1) very much; 2) somewhat; 3) never. What would be a suitable approach to express reliabilities for scales like this?

I am aware that previous questions have touched upon this topic. But I have not found any satisfying answer yet. For example, here, it was only suggested that "if you have unordered categorical data (i.e., three or more unordered categories; which you do), then you don't use Cronbach's alpha". However, no recommendation was given for what to do if you have three (ordered) categories. Any help appreciated!

Composite reliability can be thought of as a generalization of coefficient alpha, that relaxes some of the assumptions. See, e.g. https://www.statisticshowto.com/composite-reliability-definition/.

You can calculate composite reliability for ordinal measures by doing confirmatory factor analysis of a polychoric correlation matrix. Example; https://journals.sagepub.com/doi/full/10.1177/2515245920951747

Alternatively, you can use item response theory, which thinks about reliability in a different way - I like https://www.researchgate.net/publication/232569066_The_new_rules_of_measurement. Summary here: https://www.rasch.org/rmt/rmt132e.htm .

IRT considers test information, rather than reliability - and there can be different values of information at different levels of ability (or whatever is being measured). E.g. a scale might be very good at differentiating people with mild depression from people without depression, but not good at differentiating people with moderate and severe depression -the reliability of the scale varies (and reliability can be calculated for any person who has taken the test).

However, if you're going to add up the items to get a score, you might as well use coefficient alpha.

• Thanks! are you aware of sample scripts to calculate the categorical omega in Mplus per chance?
– DomB
Commented Jan 27, 2022 at 14:57
• I don't recall seeing one, but I'd be surprised if they didn't exist (I haven't used Mplus in many years). Commented Jan 27, 2022 at 17:44