Let Year 1 be last year's data and Year 2 be this year's data.

Suppose that in Year 1, you had a likert scale that was 1-9 (Categorical/Ordinal) and that in Year 2, for the same question you had a likert scale that was 1-5 (Categorical/Ordinal).

What would be some of the things that you would try (if at all) to compare the two years worth of data?

What I've done so far:

  • Compared distributions (shape, skew, and kurtosis, statistically equal)
  • Rescaled 1-9 to 1-5 and the changes YoY in frequencies match logical expectations derived from industry news/events and qualitative research findings.

Note: This is not homework. It also may not have a definite answer. But, I need a hand!

Thanks in advance!

  • $\begingroup$ why do you say Likert scale and then Categorial/Ordinal? Likert means interval scaled. Can you clarify this a little? $\endgroup$ – Henrik Feb 2 '11 at 10:36
  • $\begingroup$ To be more specific, the title should be changed to Likert "item". On your second point, I think a lot of people would disagree as to whether or not a Likert item presents interval or ordinal data. For my question, it's an agreement scale, from strongly disagree to strongly agree. Each level of agreement being a "category" and the distance between being "ordinal". But let's not get tied up in semantics! $\endgroup$ – Brandon Bertelsen Feb 2 '11 at 11:04
  • $\begingroup$ @Henrik @Brandon There were already some discussions, headed under the scales tag, about the nature and the way to treat Likert scale/item. $\endgroup$ – chl Feb 2 '11 at 19:23

This is not a complete answer; just a few points:

  • If you can administer both versions of the scale to a subsample, you could estimate what corresponding scores are on the two response formats. Then you could apply a conversion formula that is empirically justified. I can think of a number of ways of doing this. I'd be interested if anyone has an academic paper on best practice for doing this.

  • If you do a simple rescaling (1 = 1; 2 = 3; 3 = 5; 4 = 7; 5 = 9), there is no guaranty that this is justifiable. As a broad statement (at least within my experience in organisational settings) changes in item wording and changes in scale options are likely to have a greater effect on responses than any actual change in the attribute of interest. At the very least you should check whether the scale anchors used are roughly equivalent across the two response formats.

  • $\begingroup$ As a note to your second comment. The anchors are the same as they were in the previous year of the survey. Essentially, the granularity of the scale was reduced. $\endgroup$ – Brandon Bertelsen Feb 2 '11 at 9:16

[Technically you've got survey items, not Likert scales; the latter are fashioned from multiple items. See, for example, Paul Spector's Summated Rating Scale Construction {Sage}.]

The steps you take will need to depend on the audience for which you're reporting. If it's academic and rigorous, like a dissertation committee, you may face special challenges. If it's not, and if it's comfortable with the common 1-5 format, why not rescale to fit that and then report means and standard deviations (especially since shapes, skew, and kurtosis are no different from year to year. I presume the distributions are normal enough that means accurately express central tendency?).

-->Why am I treating your variables as interval-level ones? Purists may say that ordinal-level variables should not be reported via means or s.d. Well, your comments suggest, despite your use of "categorical/ordinal," that you are dealing with an ordinal level of measurement which you actually feel comfortable treating as interval-level. After all, why otherwise would you assess skewness or kurtosis. I'm guessing that your audience, too, will be ok with and will be able to relate to interval-level statistics such as means.

It sounds good that you have already explored the data graphically. If you want to go beyond assessing the magnitude of the difference and conduct an hypothesis test, why not do a T-test (independent or correlated, depending on your data) comparing the 1-5 scores pre and the 1-5 scores post, and yielding a confidence interval for the mean difference. Here I'm assuming you've got random samples from a population.

  • $\begingroup$ Yes, I realize I'm not "supposed" to be looking at some of these things for ordinal data, but really, it's the only tool I could think of to compare the two years. Really, I was looking at things that could compare the distributions. But, I guess testing means could be plausible - but a confidence interval may not necessarily include my mean as there have been many structural changes to the industry for which this question reviews YoY. $\endgroup$ – Brandon Bertelsen Feb 2 '11 at 9:11

Consider transforming the responses from both data sets into z-scores. There is going to be an ad hoc quality to any sort of rescaling but at least this way you avoid mechanically treating any particular set of intervals on one item as equivalent to any particular set on the other. I'd definitely go this route if I were using the items as predictors or outcome variables in any sort of analysis of variance. If you were doing anything w/ composite scales -- ones that aggregate likert measures -- you'd likely do essentially what I've prpposed: either you'd convert the item responses to z-scores before summing or taking their mean to form the composite scale; or you'd form a scale with factor analysis or another technique that uses the covariance matrix of the items to determine the affinity of the responses to them.

  • 2
    $\begingroup$ It appears this would force year-on-year comparisons to have equal means and variances, artificially eliminating most information about temporal change. $\endgroup$ – whuber Apr 1 '11 at 4:44
  • $\begingroup$ true. I was not thinking of comparing means of the two samples, in which case converting to z-scores is self-deating. Had covariance in mind-- e.g., assessing how one or more predictors relate to the likert item score in either yr or both combined. I tend to think covariance matrices are the only thing one should try to learn from when using likert items (people tend to invest the points on the measure with too much meaning--"but mine goes to 11"). Hope I did not throw anyone off track. $\endgroup$ – dmk38 Apr 2 '11 at 0:29
  • $\begingroup$ Agreeably some comparisons are lost. However, much of my analysis focuses on net promoter scores rather than means and variance. So, I'll try it out and see what it gets me. Cheers for the response. $\endgroup$ – Brandon Bertelsen Apr 2 '11 at 3:12

I've just had to solve this exact problem. We had a 9 point scale that was changed to a 5 point scale on a tracker going back 10 years. Not only that but some of the statements changed as well. And we were reporting as a form of Net Promoter Score.

The solution we used apply's a paired design by asking each respondent a few of the old statements the old way (as well as all the new way). We only asked a couple the old way rather than all of them since this minimises respondent fatigue. We then take each score on the 9 point scale and find it's average on the 5 point score and use this to correct for the scale change AND the statement change. This is quite similar to what is called the "Semantic Judgement of Fixed Word Value" in some papers, but instead of using experts to decide the 'word value' we used respondents actual data.

For example, if the average score on the 5 point scale was 1.2 for those respondents who answered 2 on the 9 point scale then to let us directly compare years with different scales on the 5 point scale we would replace all 2's on the 9 point scale with 1.2, then do the same for all the 9 point scores, and proceed as normal.

We did a similar thing for reporting NPS. But first we converted the 5 point scale to the NPS scale of 1 (promoter), 0 (passive), -1 (detractor) e.g. if the average on the NPS scale was 0.9 for a 2 on the 9 point scale then we replaced it with 0.9, then do the same for all the 9 point scores, and then calculated NPS normally.

To evaluate the effectiveness of this we first compared the 'uncorrected' NPS scores using the 9 and 5 point scales to see if there was actually any problem at all, and then the 'corrected' ones. I haven't got the data yet but will report back when we do!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.