Which distance to assess equality between discrete distributions?

Question

I have approximated a discrete distribution via Monte Carlo in two different ways, which metric would you use to check the distance between the two distributions? I want to use it to diagnostic whether the two methods actually are approaching the same distribution, I don't have access to the "true" distribution.

Edit to clarify: I'm simulating a CTMC many times and approximate the transition distribution with the empirical one. I simulated the MC in two ways one of which I'm not 100% sure. I know it's not a proof that the two procedures are equivalent, but at least with a large number of paths if it's not working some bells should ring

Answer 1

A commonly used measure is Kullback-Leibler Divergence (also known as relative entropy).

Given your two distributions, say $p$ and $q$, you can use KL divergence to define a distance metric to measure how “close” or “similar” they are. The divergence measure $D(p, q)$ quantifies how far $q$ is from $p$.

In your scenario, you can interpret the KL divergence as the higher the value, the more dissimilar the two distributions are. The closer the value is to 0, the more similar they are. Also note, KL divergence is non-symmetric, so your results for $KL(p || q)$ != $KL(q || p)$.

Side Note: When researching the method, you will often find that $p$ is the "true" distribution. However, in your situation, I think it is fine to use the method as a measure of how dissimilar your two distributions are.