# What to consider when choosing between f-divergence measures? (e.g.: kl-divergence, chi-square divergence, etc.)

I have some baseline population, and I have a non random sample from that population. For both the population and the sample I have observation of some measure (for simplicity, let's say age).

I would like to measure how "un-similar" my sample is from the target population, and would like to use a divergence measure (in which the $$P$$ is the distribution of age in the population and $$Q$$ the distribution of age in the sample).

I know that the f-divergence includes various known options such as kl-divergence, chi-square divergence, and others. But it's not clear to me how to prefer or decide which divergence to choose. Are there some known considerations on when using one over the other is beneficial? (in terms of interpretation, or decision making?)

For example, I know that various machine learning algorithms might use kl-divergence (or cross entropy) for optimizing, but it's not clear to me why that is better or worse then, say, chi-square divergence.

Thanks.

• Cross-entropy loss is intimately related to (and equivalent to) maximum likelihood estimation of logistic regression parameters.
– Dave
Nov 8, 2021 at 19:37
• Nov 9, 2021 at 10:31
• How are you planning to use the measure of divergence? Without that there’s not much basis for a choice. Nov 12, 2021 at 19:51
• @MattF. - you can think that I have two populations, and I wish to compare the distance of their distributions on many dimensions. That would later be used to rank dissimilarity of the two population across dimensions. The thing is that I am trying to understand how each of the distance measures might be interpreted, so to understand how each of them can be interpreted. Nov 13, 2021 at 13:07

1. The assumptions you take on $$P,Q$$ as probability measures: In Hellinger, for example, you need $$P,Q$$ to be absolutely continuous w.r.t a third probability measure $$\lambda$$. In that context, KL divergence is apparently less "demanding", only assuming $$P$$ is absolutely continuous w.r.t $$Q$$.
2. Desired divergence properties: For example, $$\chi^2$$ and KL are not symmetric, while Jensen-Shannon is.
3. The intended use of the divergence: For example, when testing $$H_0:X\sim P$$ against $$H_1:X\sim Q$$, a useful divergence would be the total variation, where $$f(x)=0.5|x-1|$$ and $$TV(P \| Q)=0.5\int{|dP-dQ|}$$. When considering a quadratic loss, the Le Cam divergence fits, where $$f(x)=0.5\left(\frac{1-x}{x+1}\right)$$ and $$LC(P \| Q)=0.5\int\frac{(dP-dQ)^2}{dP+dQ}$$. You can find out some more here (see Section 6).
• I believe your first point, regarding the Hellinger distance, is misleading. First, the Hellinger distance is an f-divergence, meaning it can be defined without reference to a third dominating measure. Second, such a measure is always guaranteed to exist -- as mentioned in your link, $\lambda = P + Q$ is one such example. Jan 25 at 2:12