When to test For Equality of Medians, and when Stochastic Equality?

Question

In statistics, we are very often interested in investigating whether some score (let's say life satisfaction) tends to be bigger in one population (let's say rich people) compared to another population (let's say poor people). Most often, this research question is formalized by testing the null hypothesis of equal means. However, for a comparison of means to be sensible, the variable of interest must be at least on an interval scale. So, for ordinal data, other formalizations should be used.

Two formalizations are generally used for ordinal data. Equality of medians and stochastic equality, which is defined as $P(X<Y)=P(X>Y)$, where $X,Y$ are the random variables representing the two populations. Many papers argue that stochastic equality is the better formalization. The core argument is that the medians can be equal even if the scores in one population are clearly bigger. Consider as an example the following mixture distributions for $X$ and $Y$. With probability .5 $X$ is just $1$, and with probability $.5$ it is sampled from $\text{Uniform}[0,1)$. Similarly, with probability .5 $Y$ is just $1$, and with probability $.5$ it is sampled from $\text{Uniform}(1,2]$. Thus, $\text{Median}(X)=1=\text{Median}(Y)$, while no realization of $Y$ is smaller than any realization of $X$ and $50\%$ of the realizations of $Y$ are bigger than all realizations of $X$.

Thus, should we stop testing for equality of medians, or are there research questions for which the equality of medians is the appropriate question? If yes, what are those?

Answer 1

This depends, in part, on the number of ordinal categories.

If the number of categories is small, then comparing the medians may be uninformative. Suppose the categories are Like/Like Somewhat/Neutral/Dislike Somewhat/Dislike, the poor group’s answers are distributed 20%-20%-20%-20%-20%, and the rich group’s answers are distributed 10%-20%-25%-25%-20%. Then both groups have a median response of Neutral, but the comparison of random pairs shows that a rich person will probably be more negative than the poor person. Depending on the size of the groups, that might be significant.

If the number of categories is large, then comparing the medians may be more useful. Suppose twelve people from groups A and B all start a task at the same time, and the only data available is that they finish in the order ABABBBAAABAB. Then a random pair from the two groups is equally likely to have the A or B person finish first, but the median person in group B finished earlier than the median person in group A. Again, with enough people in the groups, the difference in medians might be significant.

More generally, if you don’t care about small perturbations in the tails of the two grops, or if you think that there’s more noise or measurement error in those tails, then the medians are more robust and it would be appropriate to test the significance of their difference.