# What is the most effective way to combine different rating scales for the same data?

I have a collection of data which is inherently the same (would expect it to follow the same distribution) where some of it is rated with one scale of ordinal ratings and the rest of the data is rated with another scale of ordinal ratings. One scale has a larger number of possible ratings than the other. So, if I were to map from one set of ratings to the other by some sort of function (linear transformation for instance) I would get in between ratings and have to round on the output scale to get a legal discrete value. Is there a rigorously justifiable method or at least a heavily used method which allows me to analyze all of the data on a single scale whether it be the larger scale, the smaller scale, or a third new scale which is neither of the first two scales (I was thinking maybe LCM of the size of the two scales if it is not too big)?

• Are you treating these ordinal results as categorical or continuous in your main analysis? Are your samples obtained using simple random sampling, or is it a retrospective or stratified sampling mechanism? What are the two scales you're referring to? – AdamO Feb 13 '12 at 16:53
• The ordinal data is a set of equally spaced integers. We can say scale 1 is {a1, a2, ... am} and scale 2 is {b1, b2, ... bn} where again all entries in both sets are equally spaced integers for example {3,4,5,6,7} and {11,12,13,14,15,16}. I would need a single way to represent all of the data. The data would not be 2 subsets from the same experiment, rather it constitutes 2 different experiments conducted measuring the same phenomenon. – demongolem Feb 13 '12 at 17:15
• I already understand this much. What I am asking, first, is what scales are these? CBCL? ECOG? In terms of "representing the data" are you talking about using a graphical or tabular presentation of numbers and frequencies or are you talking about something like adjustment in a regression model? If the latter, how exactly are you intending to treat such data? Would you use indicator variables for the various categories, or would you treat the ordinal scale as a continuous variable with values given by its index? – AdamO Feb 13 '12 at 18:56
• This is a supervised learning setting. I am training on these two different experiment datasets to in the future predict what the rating should be (so having only 1 methodology for rating on test data is good). As example, yesterday I was using WEKA because .arff is one format I have the data in. I am setting the ranking to be "class" so I was telling WEKA nominal/categorical data. I was trying to do something like logistic regression or SVM (the two best models currently on the data when I consider data for only a single rating). My nine features used to determine ranking are numerical. – demongolem Feb 13 '12 at 20:15
• I think I get what you say by ECOG and CBCL. ECOG, ecog.org/general/perf_stat.html, is much how my ratings are given, yes. It is a bit of criteria, which may be subjective partially, to assign a rating. – demongolem Feb 13 '12 at 20:17

this appears to be an equating problem, common in field of educational measurement. kolen and brennan have written an entire text on this subject, for which some of the main issues are in regard to the comparability of the examinee groups.

earlier commenters have suggested the use of item response theory (IRT) methods (i.e., PCM, RSM, and Rasch). these analyses are all appropriate depending on the consideration of some fundamental assumptions about your scales and data. one of the usual outcomes of an IRT analysis is includes the production of values for each examinee/subject commonly referred to as "theta" - a continuous variable representing the quantity of the hypothesized construct possessed by examinees/subjects. these theta values follow the same scale regardless of the sample or instrument used. however, IRT is not the first step that i would recommend here.

i would first suggest a method grounded in classical test theory known as equipercentile equating. quite simply, the cumulative distribution function (CDF) of scores on A are converted via an equating function such that they are equal to the CDF of scores on B. this method does not require the implicit consideration of measurement error or the population distribution of the hypothesized construct, as is the case in IRT. the latter methods, however, do provide a wealth of additional diagnostic information.

• I was not able to immediately get my hands on kolen and brennan, however I found vonDavier "Statistical Models for Test Equating, Scaling, and Linking" also by Springer. From what I have read so far, it looks like a valuable read for me. – demongolem Feb 16 '12 at 1:04

I would recommend to start with a Partial Credit Model (PCM; Master, 1982). The PCM will accommodate the variation of the different rating scales without problem. Notice however, that the key feature of the PCM does not assume that the 'distances' between the steps in the scales are common across items, allowing the size of the 'steps' to vary from item to item.

A second more stringent step would be to use the Rating Scale Model (RSM; Andrich, 1978) which would allow you to treat both scales as 'true' rating scales (i.e., the size of the 'steps' are shared across the items) as long as you parameterize the two scale separately.

For a brief summary of both these models you can see the wikipedia page on polytomous Rasch models

Unless you have powerful theoretical reasons to prefer the more stringent model (the RSM), I would suggest that you try using the PCM because it should be easily estimable with R (using the eRm package for example) or Stata or with dedicated psychometrics software such as WinSteps or Conquest.

• the PCM and generalised PCM can also be estimated using the ltm package, if you encounter a poor fit using the Rasch PCM – richiemorrisroe Feb 14 '12 at 12:08

You could do a separate Rasch analysis of each of the two scales to see whether the item scores are what you expect. The analysis would provide information on the relative difficulty of items, in logits, so you could compare whether items are truly equivalent between scales by whether they have the same logit values. In addition, the output will include category probability curves, which would be useful in determining whether the response picture for each theoretically equivalent item is the same. You may find that some response categories which have a wider scale have no probability of being selected, and the graphs will suggest which categories can be combined. There are a lot of diagnostics that can be performed in Rasch analysis, I have only mentioned two of the most helpful here, with another important output being item fit.