How to compare Likert scales with varying number of categories across time?

Question

Let Year 1 be last year's data and Year 2 be this year's data.

Suppose that in Year 1, you had a likert scale that was 1-9 (Categorical/Ordinal) and that in Year 2, for the same question you had a likert scale that was 1-5 (Categorical/Ordinal).

What would be some of the things that you would try (if at all) to compare the two years worth of data?

What I've done so far:

• Compared distributions (shape, skew, and kurtosis, statistically equal)
• Rescaled 1-9 to 1-5 and the changes YoY in frequencies match logical expectations derived from industry news/events and qualitative research findings.

Note: This is not homework. It also may not have a definite answer. But, I need a hand!

Thanks in advance!

Answer 1

This is not a complete answer; just a few points:

• If you can administer both versions of the scale to a subsample, you could estimate what corresponding scores are on the two response formats. Then you could apply a conversion formula that is empirically justified. I can think of a number of ways of doing this. I'd be interested if anyone has an academic paper on best practice for doing this.

• If you do a simple rescaling (1 = 1; 2 = 3; 3 = 5; 4 = 7; 5 = 9), there is no guaranty that this is justifiable. As a broad statement (at least within my experience in organisational settings) changes in item wording and changes in scale options are likely to have a greater effect on responses than any actual change in the attribute of interest. At the very least you should check whether the scale anchors used are roughly equivalent across the two response formats.

Answer 2

[Technically you've got survey items, not Likert scales; the latter are fashioned from multiple items. See, for example, Paul Spector's Summated Rating Scale Construction {Sage}.]

The steps you take will need to depend on the audience for which you're reporting. If it's academic and rigorous, like a dissertation committee, you may face special challenges. If it's not, and if it's comfortable with the common 1-5 format, why not rescale to fit that and then report means and standard deviations (especially since shapes, skew, and kurtosis are no different from year to year. I presume the distributions are normal enough that means accurately express central tendency?).

-->Why am I treating your variables as interval-level ones? Purists may say that ordinal-level variables should not be reported via means or s.d. Well, your comments suggest, despite your use of "categorical/ordinal," that you are dealing with an ordinal level of measurement which you actually feel comfortable treating as interval-level. After all, why otherwise would you assess skewness or kurtosis. I'm guessing that your audience, too, will be ok with and will be able to relate to interval-level statistics such as means.

It sounds good that you have already explored the data graphically. If you want to go beyond assessing the magnitude of the difference and conduct an hypothesis test, why not do a T-test (independent or correlated, depending on your data) comparing the 1-5 scores pre and the 1-5 scores post, and yielding a confidence interval for the mean difference. Here I'm assuming you've got random samples from a population.

Answer 3

Consider transforming the responses from both data sets into z-scores. There is going to be an ad hoc quality to any sort of rescaling but at least this way you avoid mechanically treating any particular set of intervals on one item as equivalent to any particular set on the other. I'd definitely go this route if I were using the items as predictors or outcome variables in any sort of analysis of variance. If you were doing anything w/ composite scales -- ones that aggregate likert measures -- you'd likely do essentially what I've prpposed: either you'd convert the item responses to z-scores before summing or taking their mean to form the composite scale; or you'd form a scale with factor analysis or another technique that uses the covariance matrix of the items to determine the affinity of the responses to them.