How to test for differences between medians of multiple Likert items?

Question

In a questionnaire study, we asked respondents to express their attitudes towards how different winter climate factors such as snow, slipperiness might affect their choice to walk and cycle to work. The sample composed of 500 individuals and answers were in form of 5 scales rating form very negative to very positive (ordinal scale).

If I want to compare the responses to different questions, I guess median is a proper tool since the data is ordinal. I know that to compare means there are different statistical tests to show if the probability of difference is significant (t-test or non- parametric test..). But I am a bit confused if I can use these test on the type of data I explained here.

• Is there a test statistics to use for comparing medians?
• Or I should transfer data to interval scale if it is appropriate?

Answer 1

I find the mean to be a much more useful indicator of central tendency of Likert items than the median. I have elaborated on my argument here on a question asking about whether to use the mean or median for likert items.

A recap of some of these reasons:

• The mean is more informative; the median is too gross for Likert items. For example, the median of 1 1 3 3 3 is the same as 3 3 3 5 5 (i.e., 3) but the mean reflects the difference.
• Likert items are often phrased in ways where the equal distance between categories assumption is a useful starting point.
• Even if individual responses are discrete, the group level measurement approaches continuity (with 500 people and a 5 point scale, the value of your mean could take on 500 * 4 + 1 = 2001 different values)
• There is little argument that a percentage is a useful summary in yes-no type questions (e.g., voting). This is just the mean where responses have been coded 0 and 1. Treating a 5 point likert scale as 1 2 3 4 5 seems almost as natural to me.
• Other plausible scalings of the Likert items probably wont change inferences substantively regarding whether differences between means exist (but you can check this).

If you are persuaded that the mean is the appropriate measure of central tendency, then you would want to structure your hypothesis tests so that they test for differences between means. A paired sample t-test would allow for a pair-wise comparison of means, but there would be issues around the accuracy of p-values given the discrete and non-normal error distribution. Nonetheless, adopting a non-parametric approach is not a solution, because it changes the hypothesis.

I would expect that the paired sample t-test would be fairly robust at least for typicaly Likert item means that avoid either extreme on the scale, but I don't have any simulation studies on hand.

Answer 2

Generally, I agree with Jeromy's arguments that the mean is a reasonable statistic for Likert scales. What could speak for the median, is that the median is a much more robust measure of location as it protects against outliers (it has the highest possible breakdown point of 50%). However, as Likert scales are bounded scales, the possibility of extreme outliers is very low (only if your data is extremely skewed). Furthermore, the median usually trims too much from the data, so you could consider using trimmed means instead. An amount of 20% trimming usually is recommended [1].

If you want to calculate a paired test of the difference of medians, I would recommend to compare the means using a percentile bootstrap method (this is the only method for comparing medians that works well in the case of tied values, see Wilcox, 2005 [1]).

In the WRS package for R, there is a function called trimpb2 which does this calculation for two independent samples (you can also calculate a p value for trimmend means with that function). In your case, however, you need to compare dependent groups. In this case, you can also do a bias-adjusted percentile bootstrap method [2].

Note, however, that the difference of the medians of the marginal distributions is not the same as looking at the median of the difference scores. The first answers the question 'How does the typical response from the first group differ from the second' and is performed by the WRS function rmmcppb. The second answers the question 'What is the typical difference score' and is performed by the WRS function rmmcppbd.

[1] Wilcox, R. R. (2005). Introduction to robust estimation and hypothesis testing. San Diego: Academic Press.

[2] Wilcox, R. R. (2006). Pairwise comparisons of dependent groups based on medians. Computational Statistics & Data Analysis, 50, 2933-2941. doi:10.1016/j.csda.2005.04.017

Answer 3

One option for comparing medians is permutation tests. However, if you are comparing the answers 2 questions that were filled out by the same set of people (paired data) then you might also want to look at McNemar's test and the variations on it.

To exapnd a bit, the idea of the McNemar test (and extensions of it) is to look at a matrix with the counts of how many respondents chose the combinations, so an idividual would contribute to the count in the cell whose column is determined by their answer to question 1 and row is determined by their answer to question 2 (table or crosstable commands create the matrix). The pattern in this matrix will probably be more informative than a simple mean or median. The diagonal represents people who responded the same to the 2 questions, the upper triangle are those that responded higher on the 1st question than the 2nd question, and the lower triangle the difference. The distance from the diagonal indicates how different the 2 answers were. Variations on the McNemar test look at whether the counts in the 2 triangles are different, or if the matrix is symmetric. To take into account the ordinal (vs. nominal) nature of the data the distance from the diagonal is taken into account.

Just looking at the patterns in the table may be enough for your purposes, but if you need a formal test, then you can either go with the suggested tests, or do some form of permutation test (exactly how depends on what you are looking for or trying to show).