How to compare the distributions of two variables

Question

The attached figure plots the distributions of two variables.

I want to demonstrate statistically how closely the the two distributions match each other. What is the best way of doing this?

Answer 1

There might not be a best solution as such. But if you are trying to visualize the distribution similarity, you can use violin plots instead of box-plots.

However, if statistical measures of similarity is what you are trying to calculate, then I think you should try fitting the distributions against several standard distributions and see how well they compare. If you are familiar with GLDs (Generalized Lambda Distributions), you can obtain GLD fits for each of the distributions and compare their resulting parameters.

Answer 2

To compare two distributions you have several options.

First of all you can visualize the two distributions:

• plot the probability density functions
• plot the cumulative distribution functions
• a quantile-quantile plot

Then, you can statistically characterize the differences between the two distributions:

• run statistical hypothesis tests to check whether there is a significant difference between the two samples; in your case (distributions seem to be a bit skewed) you should rely on non-parametric Mann-Whitney or Kolmogorov-Smirnov test (if you don't know how such tests work, you can find nice explanations on CrossValidated)
• compute Kullback–Leibler divergence (and similar measures)

In particular, Kullback-Leibler divergence is a non-symmetric measure of the difference between two probability distributions, say $P$ and $Q$. More precisely, the KS divergence of $Q$ to $P$, $D_{KL}(P||Q)$, is a measure of the information lost when $Q$ is used to approximate $P$.

For discrete probability distributions $P$ and $Q$, the KS divergence is defined as: $$D_{KL}(P||Q) = \sum_{i}P(i) \ln \frac{P(i)}{Q(i)}, $$ whereas in the continuous case: $$D_{KL}(P||Q) = \int_{-\infty}^{\infty}p(x) \ln \frac{p(x)}{q(x)}dx,$$

where $p$ and $q$ represent the densities of $P$ and $Q$.

Note that the KS divergence is always non-negative: $D_{KL}(P||Q) \geq 0$, and the equality holds if and only if $P = Q$ almost everywhere.

Answer 3

Check out the Kullback-Leibler divergence (http://en.wikipedia.org/wiki/Kullback_Leibler_divergence). This describes the loss of information when one distribution is used to approximate the other