I'm trying to fashion survey experiments for which the demographic composition of the samples needs to be as close as possible to a 'golden' composition. For concreteness, let's say the 'golden' composition is 14% middle class, 35% upper-middle class, and 51% upper class.

I have a whole bunch of samples, each with slightly different compositions. For example:

S1 = 10% middle class, 41% upper-middle class, 49% upper class
S2 = 33% middle class, 33% upper-middle class, 34% upper class

I'm trying to find a good statistical measure to describe how close the distribution in each sample is to the ideal, 'golden' composition. Is there a simple metric here that I could use?


What about Kullback–Leibler divergence? This is a good measure (in terms of usability in the classification based machine learning) of differences between two probability distributions. This is an asymmetric measure, but as you are comparing many compositions with one golden standard it seems quite a good choice.

$$D_{KL}(P||Q) = \sum_i \ln \left ( \frac{P(i)}{Q(i)} \right ) P(i) $$

It is the expectation of the logarithmic difference between the probabilities P and Q, where the expectation is taken using the probabilities P.

It can be also treated as a difference in the entropy

$$D_{KL}(P||Q) = H(P,Q) - H(P)$$

where $H(P,Q)$ is a cross entropy and $H(P)$ is entropy.

  • $\begingroup$ To confirm, it is okay to use K-L divergence with categorical distributions, correct? For example, if I had another golden standard like 55% from New Jersey, 5% from California, 40% from New York? $\endgroup$
    – user139014
    Aug 12 '13 at 7:59
  • 1
    $\begingroup$ Yes, this is often used measure for eg. computing how sure is ensamble of models about their multi class prediction (each gives its own $P(y|x)$ for categorical $y$'s, and $D_{KL}$ is used to measure if they reached a consensus. $\endgroup$
    – lejlot
    Aug 12 '13 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.