Appropriate divergence measure for a distribution over ordinal values

Question

I would like to measure the divergence (or, more appropriately, symmetric difference) between two distributions $P$ and $D$. In general, you could consider using a measure like Jensen-Shannon divergence (JSD). However, consider the case in which we are attempting to judge our predictions of the distribution of scores for an olympic diver. If the scores are out of 5, and the true distribution is $[0, 0, 0, 1.0, 0]$, then a guess of $[0, 0, 0, 0, 1.0]$ is much better than a guess of $[1.0, 0, 0, 0, 0]$. However, to a measure like JSD or Kullback–Leibler, these guesses are equally bad, since the distribution is assumed to be over categorical outcomes as affirmed here.

Are there any accepted ways of comparing these distributions given the assumption that the distribution is over items that can be considered to have an ordering? Simply smoothing the predictions would have this effect to some degree, but that would require adding more parameters (i.e., smoothing strength) and I'm not sure how valid it would be.

Answer 1

The earth mover distance could do the trick. Informally, it considers distributions as a piles of dirt. The dirt can be moved, with an associated cost given by the amount of dirt moved times the distance it's moved. The distance from distribution A to distribution B is the minimum possible cost of moving dirt to transform A into B.

Answer 2

One paper you can check is https://dl.acm.org/citation.cfm?doid=2911451.2914749 ; the authors indeed use the Earth Mover's Distance for estimating how well a true ordinal distribution is estimated by a predicted distribution.