Measures like KL divergence can be symmetrized (into JS divergence). Bhattacharyya distance serves a similar function. Either is well-suited to both continuous distributions and discrete (e.g. multinomial) distributions. Is there a measure that conveys the divergence or distance between ordered distributions? One that penalizes more if probability mass is in a more distant category?

The particular application is comparing populations' responses to a Likert-scale survey question. I have multiple paired populations, and I treat each population's response as a probability distribution over {1,...,7}. I want to compare the pairs' distances (or symmetric divergences)—e.g., are A1 and A2 farther than B1 and B2? With a cursory understanding, I'm drawn to the Wasserstein distance, but perhaps there is something more standard or appropriate.

  • $\begingroup$ The earth mover's distance (aka Wasserstein metric) immediately jumps out as something to try. Even though it's less widely used than some other distance metrics, it's certainly established/known. So, you may have already answered your own question on that front. $\endgroup$
    – user20160
    Commented Feb 26, 2018 at 1:40
  • $\begingroup$ That's good to know. I know it's become popular as an objective function for GANs since directly optimizing KL in those cases is tricky. Do you want to write that up as an "answer"? I'll happily accept it to help future users. $\endgroup$ Commented Feb 26, 2018 at 14:06
  • $\begingroup$ I'm probably not seeing how Wasserstein distance (EMD) fully accounts for the ordinal semantics. Imagine you have a lot of mass on the edges and little in the middle of a distribution. Now assume two additional distributions, one where the mass in the middle is filled at the expense of the left side of the distribution and one where it is filled at the expense of the right side. Would they not get the same Wasserstein distance from the original first distribution? $\endgroup$
    – matanox
    Commented Sep 28, 2018 at 20:10


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.