What to consider when choosing between f-divergence measures? (e.g.: kl-divergence, chi-square divergence, etc.)

Question

I have some baseline population, and I have a non random sample from that population. For both the population and the sample I have observation of some measure (for simplicity, let's say age).

I would like to measure how "un-similar" my sample is from the target population, and would like to use a divergence measure (in which the $P$ is the distribution of age in the population and $Q$ the distribution of age in the sample).

I know that the f-divergence includes various known options such as kl-divergence, chi-square divergence, and others. But it's not clear to me how to prefer or decide which divergence to choose. Are there some known considerations on when using one over the other is beneficial? (in terms of interpretation, or decision making?)

For example, I know that various machine learning algorithms might use kl-divergence (or cross entropy) for optimizing, but it's not clear to me why that is better or worse then, say, chi-square divergence.

Thanks.

Answer 1

I think you should consider three factors when choosing a divergence:

1. The assumptions you take on $P,Q$ as probability measures: In Hellinger, for example, you need $P,Q$ to be absolutely continuous w.r.t a third probability measure $\lambda$. In that context, KL divergence is apparently less "demanding", only assuming $P$ is absolutely continuous w.r.t $Q$.

2. Desired divergence properties: For example, $\chi^2$ and KL are not symmetric, while Jensen-Shannon is.

3. The intended use of the divergence: For example, when testing $H_0:X\sim P$ against $H_1:X\sim Q$, a useful divergence would be the total variation, where $f(x)=0.5|x-1|$ and $TV(P \| Q)=0.5\int{|dP-dQ|}$. When considering a quadratic loss, the Le Cam divergence fits, where $f(x)=0.5\left(\frac{1-x}{x+1}\right)$ and $LC(P \| Q)=0.5\int\frac{(dP-dQ)^2}{dP+dQ}$. You can find out some more here (see Section 6).