Testing for differences in distributions (medians AND dispersion) between groups with a Likert scale question

Question

I am interested in testing for significant differences with a Likert scale question between three groups. There are plenty of questions about comparing medians here, but I am interested in differences in both medians and dispersion - basically any difference in the distributions.

For example:

Research question: Are there significant differences between groups in the level of agreement with the following statement: "I enjoy playing video games"?

Groups: Black, Hispanic, White

Possible responses: Strongly disagree, disagree, neutral, agree, strongly agree

My initial thought was to use the Kruskal-Wallis test first, but this site states that the test assumes that the groups have the same shape and variability (which would also mean that Kruskal-Wallis isn't capturing differences in dispersion? Only medians?). Upon reading this, I thought I ought to test for dispersion first using the Brown-Forsythe test.

My questions are thus:

(1) Is my proceeding appropriate? Should the Brown-Forsythe test (or some reasonable alternative) always be used prior to Kruskal-Wallis (when using ordinal variables) to ensure equal dispersion?

(2) If I only had two groups, Mann-Whitney would be more appropriate from my understanding. This site also says that an appropriate alternative hypothesis for the Mann-Whitney test is that 'the distributions between groups are different.' Given this, would I still need to test for dispersion differences between groups first? If the Kruskal-Wallis test is an extension of Mann-Whitney for more than two groups, can the assumptions be different?

(3) Is there a different test that I should be using? I know that there are some tests designed specifically for comparing distributions, but these seem to have some shortcomings based on my needs. For example, Komolgorov-Smirnov requires continuous data, Chi-square doesn't take into account the ordinal nature of data (so it's not as powerful?), etc.

Note(s):

I know that there is considerable debate about whether Likert scale questions should be assumed interval or ordinal, but for this question can we just assume that they're ordinal? :)

Answer 1

I am interested in differences in both medians and dispersion

The Kruskal-Wallis test doesn't test for differences in medians without one additional assumption or another.

this site states that the test assumes that the groups have the same shape and variability

No it doesn't say that (and a good thing too, or it would be wrong if it did).

It says something slightly different:

*In order to know how to interpret the results ... *

This is not the same as saying that the test assumes it.

Here's the actual situation: If you assume that the only possible differences are location shifts then a difference in location by the measure the Kruskal-Wallis picks up is equally a difference in any measure of location (as long as it exists) -- it's just as much a difference in medians or of means, for example. That's a nice situation to be in, but it's not a requirement.

And a good thing, too, otherwise you could pretty much never use it on data like Likert-scale items. Consider that you have one group distributed across all options. If the only possible alternatives are shifts and the only possible values taken by a Likert-type item are categorical, everything would have to shift up or down by a category. If you already take up the width of the scale, that's impossible.

There are differences that don't just correspond to location shifts that are perfectly easy to interpret. For example, imagine a variable like "time to complete a task". Changes in that might readily be multiplicative rather than additive (a task might take 10% more time or 5% less time, as conditions change for example - this is a stretching or compression). If present, such changes of scale rather than of location would be detected by a Kruskal-Wallis and are no harder to interpret than location shifts.

One problem you can have with a Kruskal-Wallis is it's possible for there to be pairwise differences that lead to a cycle: A tends to be larger than B (specifically $P(A>B)>\frac12$, B larger than C and C larger than A.

Now that situation is tricky to interpret. (However, if you're interested in any differences, this additional difficulty shouldn't bother you in the least, because it still indicates a difference in distributions.)

By assuming that the only differences are location shifts, you can certainly avoid this problem ... if the differences are all shifts of the whole distribution up or down, there can be no cycle. But that's much too restrictive a restriction if that's what they're trying to avoid.

mean that Kruskal-Wallis isn't capturing differences in dispersion?

It can detect differences in dispersion that change the probability that a value from one of the variables exceeds a value from another -- such as the scale-shifts I mentioned before, for example. But it doesn't detect changes in dispersion that leaves that probability at $\frac12$.

Only medians?

As I said above it's not actually a test of medians, but it does detect location shifts. Note that the scale shift I mentioned would also lead to a difference in medians (as well as a difference in dispersion).

--

Note that you say this:

medians and dispersion - basically any difference in the distributions.

No, changes to location and dispersion don't account for all possible changes - its trivial to find differences that are not changes to location and spread for typical measures of each.

If you really mean you want to test for any differences, even those that might not be location and dispersion, there are k-sample versions of the Komogorov-Smirnov test and the Cramer-von Mises test that would be suitable. (Or you could do a chi-square, for example, but the others will be more powerful against the sort of differences it sounds like you might like to look for)

On the other hand, if you mean that differences other than location and dispersion are of little interest, there are nonparametric tests that are sensitive to both shifts at once, such as Rosenbaum's test in the two-sample case (which can pick up when there's an increase in location accompanied by increase in spread -- but the Kruskal-Wallis should manage that particular alternative pretty well in any case).

--

If neither the Kruskal Wallis not the k-sample Kolmogorov-Smirnov appeal, I'd suggest considering a different strategy, which is based on partitioning the chi-square I mentioned before, into orthogonal components (linear, quadratic, cubic and quartic). The first two components would be the ones to test if you're interested mainly in location and scale differences.

Answer 2

Try Hollander and Wolfe, 1973, Nonparametric Statistics, the chapter on detecting differences in dispersion. Ansari-Bradley test, Moses test, and Miller’s Jacknife.