Is the difference between two ordinal survey responses still ordinal?

  • 2
    $\begingroup$ This regards not the question itself but some answers. The line taken seems to be to say that in analysis of ordinal data operations such as subtracting, averaging etc. should not be used. I just want to point out that this is controversial, see, e.g., Velleman, P.F. and Wilkinson, L. (1993) Nominal, Ordinal, Interval and Ration Typology Are Misleading. The American Statistician, 47, 65-72. doi.org/10.1080/00031305.1993.10475938. One argument is that assumptions of statistical procedures are about distributions of values rather than the meaning of the data. $\endgroup$ Commented Aug 18, 2022 at 10:38
  • 3
    $\begingroup$ Hand, D. J. (1996). Statistics and the Theory of Measurement. Journal of the Royal Statistical Society. Series A (Statistics in Society), 159(3), 445–492. doi.org/10.2307/2983326 with discussion and a later book on measurement by Hand cover many aspects of the debate with balanced conclusions. One implication relevant to the question is that often supposedly ordinal data are not purely ordinal (holding somewhat stronger information not preserved by all monotonic transformations), in which case subtraction can be at least empirically meaningful. $\endgroup$ Commented Aug 18, 2022 at 10:42
  • 3
    $\begingroup$ @TinaHeeley There are many types of data that are not really interval but more than ordinal. For example age categories and other supposedly ordinal variables carry somewhat incomplete quantitative information. Ranks have a numerical meaning even though this may differ from what is ranked. If Likert scales in questionnaires are presented with a numbering of categories, respondents may assume that these are numerically evaluated and interpreted, which may influence their response. $\endgroup$ Commented Aug 18, 2022 at 15:15
  • 1
    $\begingroup$ Marks or grades given by teachers or academics are a frequent example that should be familiar. There is no independent check on whether someone getting 8/10 on a test is twice as good or as strong in terms of anything but the marking scheme as someone getting 4/10. The same even goes for essays, dissertations or theses graded on a more nuanced scale, e.g. even down to a resolution of 1%. Yet at many institutions working with averages or other summaries is expected and the problems with that are mostly not to do with what kind of measurement scale is being used. $\endgroup$
    – Nick Cox
    Commented Aug 19, 2022 at 15:30
  • 1
    $\begingroup$ (ctd) In short, by the high standards of measurement theorists such measurement is ordinal at best, but in practice it's often treated as interval. Tukey and others have underlined that -- although temperature is routinely regarded as measured on an interval or ratio scale -- for most of its history temperature measurement was based on all kinds of rough or indirect approximations. Even now length of mercury thread in a glass tube has no bearing on other ways to measure or estimate temperatures in many circumstances, e.g. in astronomy. $\endgroup$
    – Nick Cox
    Commented Aug 19, 2022 at 15:34

4 Answers 4


Clearly not, in general. Take a 4-level pain scale for example (none, mild, moderate, severe). Going from moderate to severe pain may be far worse than going from mild to moderate pain. Yet they both have a difference of 1 if pain were coded 0,1,2,3.

Ordinal scale data need to be analyzed in a way that subtraction is not used. This is discussed further here.

  • 1
    $\begingroup$ Thank you Frank! I have 2 separate variables (y1, y2) at a single timepoint. My variables might reasonably approximate interval spacing (more so than pain) and have a 7-point scale. I think my analysis options are: 1. Assume interval spacing, treat the difference (y1-y2) as continuous, use linear regression 2. Categorise the difference as >, <, or equal to and use ordered logistic regression 3. Do not calculate difference, use y1 independent variable, condition on y2, and use ordered logistic regression Any thoughts on the merits of each option or any alternatives would be appreciated. $\endgroup$ Commented Aug 17, 2022 at 15:53
  • $\begingroup$ Some variation on the rank difference test may be in order: see here $\endgroup$ Commented Aug 17, 2022 at 20:01
  • $\begingroup$ I am not familiar with Kornbrot's rank difference test and it doesn't appear to have a Stata package. Why is it preferable to the Wilcoxon signed-rank test for my data specifically? Also, is there an analogous regression model to allow adjustment for covariates? $\endgroup$ Commented Aug 18, 2022 at 11:21
  • 1
    $\begingroup$ I don't know that it's been extended to covariates. But the advantage is that it is invariant to how you transform Y and doesn't assume symmetry of a distribution of change scores as the Wilcoxon signed-rank test does. $\endgroup$ Commented Aug 18, 2022 at 12:10
  • $\begingroup$ Wilcoxon signed-rank starts with subtracting one from the other. $\endgroup$
    – Bernhard
    Commented Aug 18, 2022 at 15:12

If you are taking the difference of two ordinal responses, you are not treating the responses as ordinal, but instead treating them as if they were interval.

This isn't always appreciated.

One example is with the Wilcoxon signed rank test. the procedure begins by subtracting the paired response values, so isn't applicable for strictly ordinal data.

My understanding is that aligned ranks transformation anova procedure also begins by subtracting values.

I have seen people argue that data from scales created by summing several Likert-type items should be treated as ordinal. But I would argue that once you are summing values, you are already treating the responses from the Likert-type items as interval.

EDIT: Based on some discussion in the comments, here is an example of using ordinal regression with unstructured thresholds for cut points, with a repeated measures design. In R. (Can also be run at rdrr.io/snippets/.)


Respondent = factor(c(letters[1:12], letters[1:12]))
Time       = factor(rep(c("Before", "After"), each=12), 
    levels=c("Before", "After"))
Score      = as.factor(c(1,2,3,1,2,3,1,2,3,4,4,4,
xtabs(~ Time + Score)

   ###         Score
   ### Time     1 2 3 4 5
   ###   Before 3 3 3 3 0
   ###   After  0 4 4 2 2

model = clmm(Score ~ Time + (1|Respondent))

   ###           Estimate Std. Error z value Pr(>|z|)   
   ### TimeAfter   2.7262     0.9758   2.794  0.00521 **


   ### model term df1 df2 F.ratio p.value
   ### Time         1 Inf   7.805  0.0052

wilcox.test(as.numeric(Score) ~ Time, paired=TRUE)

   ### Wilcoxon signed rank test with continuity correction
   ### V = 4.5, p-value = 0.02475
  • 1
    $\begingroup$ Thank you Sal. I understand that the Wilcoxon signed rank test is a special case of ordinal logistic regression model. Given that it is widely used on ordinal data, though it is not strictly applicable as you say, it is therefore reasonable to use the subtracted paired response values as the dependent variable in an ordinal logistic regression? $\endgroup$ Commented Aug 18, 2022 at 11:36
  • $\begingroup$ Whether it's reasonable is up to you. If you are subtracting values, you are treating them as interval rather than strictly ordinal. ... At least in some software implementations, you can analyze a repeated measures design with ordinal regression, without assuming equal spacing between the categories. $\endgroup$ Commented Aug 18, 2022 at 13:20

The only mathematical relations that exists between ordinal data is "greater than", "less than", and "equal". Any other mathematical relation, such as addition, subtraction, multiplication, etc., marks the data as being treated as a more complex data type such as interval or ratio data.

I think this quote from Frank Harrell's answer bears addressing:

Take a 4-level pain scale for example (none, mild, moderate, severe). Going from moderate to severe pain may be far worse than going from mild to moderate pain. Yet they both have a difference of 1 if pain were coded 0,1,2,3.

In ordinal data, the "separation" between different values isn't merely "not necessarily constant", it's not defined. The levels are being treated as having no defined metric. The numbers are merely labels that have a particular order. You could just as easily call the pain levels "banana, orange, apple, pear", as long as it's understood that the labels have that order. Saying "pear minus apple might be larger than apple minus orange" is meaningless. From a CS perspective, the pain level are objects with the methods .lt(), .gt(), and .eq(), .repr()and nothing else. There is no .minus() or .plus() method. Pain level "0" is just an object whose .repr() value is "0". It has no other relation to 0. Is it not the int 0, and you can't apply int operations to it.

  • 1
    $\begingroup$ This is overstating the extent of their incomparability. In the four-point pain scale example, you can’t compare the difference 0–1 with the difference 2–3 — but you can say some meaningful things derived from the original ordering, e.g. that the difference 1–4 is bigger than the difference 2–3. (Mathematically, this is the fact that the formal differences between elements of an ordered set form a natural partial order.) $\endgroup$
    – PLL
    Commented Aug 19, 2022 at 13:20

Ordinal variables are often not even numeric - the difference between ordinal variables isn't even a defined operation, much less an ordinal variable itself.

Consider three hot sauces, mild, spicy, and extra hot. How do you define (spicy - mild) or (extra hot - spicy)? There is no rationale in mapping these ordinal values to numbers and treating them as numeric variables.

If you had a rationale for claiming that the difference between mild and spicy is smaller than the difference between spicy and extra hot, you are now quantifying the differences between levels in a way that defies the definition of an ordinal variable, which indicates that you can only rank the levels but not comment on the "distance" between them. If you can quantify the size of the difference between levels, it's not an ordinal variable in the first place.

  • 1
    $\begingroup$ Your example is convincing as your example. The problem is that the label ordinal has also been applied to quite different kinds of measurement (or "measurement"). $\endgroup$
    – Nick Cox
    Commented Aug 19, 2022 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.