2
$\begingroup$

I'm currently studying about the two methods in the title.
Does the Bhattacharyya distance and KL divergence measure the same thing, but in a different way?
I know the things that Bhattacharyya distance and KL divergence take into account when calculating is different, but in their essence, do they both measure difference between 2 probabilities distributions?
Thanks in advance

$\endgroup$
3
  • $\begingroup$ KL divergence isn’t a distance. $\endgroup$ – Sycorax Mar 29 at 13:00
  • $\begingroup$ @Sycorax but that's, as a concept (of course you can't compare them on equation), the same thing, right? $\endgroup$ – Dave Mar 29 at 13:26
  • 1
    $\begingroup$ I suppose they measure the same thing in the same sense that swimming speed and bench press ability measure athleticism. // KL divergence is not symmetric and thus is not a metric, so calling it a distance (instead of a divergence) is iffy. (This is a different "Dave" posting.) $\endgroup$ – Dave Mar 29 at 13:31
5
$\begingroup$

Bhattacharyya distance and KL divergence both measure the dissimilarity between two probability distributions, but they do it in different ways. This means their numerical values are not directly comparable and, furthermore, they may not agree on the relative dissimilarities between multiple distributions. For example, suppose we have two distributions $q_1, q_2$ and want to determine which is closer to a reference distribution $p$. We might find that $q_1$ is closer to $p$ in terms of Bhattacharyya distance, but $q_2$ is closer in terms of KL divergence (or vice versa).

Bhattacharyya distance and KL divergence are both divergences, meaning they always take non-negative values, with a value of zero if and only if the two distributions are identical. However, neither is a true distance metric, which must additionally satisfy the triangle inequality (which neither measure does) and be symmetric. Bhattacharyya distance is symmetric ($D_B(p,q) = D_B(q,p)$), whereas KL divergence is not ($D_{KL}(p \parallel q) \ne D_{KL}(q \parallel p)$). The meaning of $D_{KL}(p \parallel q)$ is different than $D_{KL}(q \parallel p)$, and it matters a great deal which one we use.

$\endgroup$
2
  • $\begingroup$ Thank you for your explanation. Do you have any general rules for choosing Bhattacharyya distance over KL divergence, and vice versa? When is the best to use each? Of course it is just a general guideline and each case has its special properties. $\endgroup$ – Dave Mar 29 at 16:33
  • 1
    $\begingroup$ @Dave Glad to help. I doubt there are any general rules for when to use each; it will come down to the specific problem you're working on. Sometimes it's just a matter of convenience, and you pick whatever's easiest to work with. Other times, one cares more about differences in their behavior; this would be best suited for a followup question $\endgroup$ – user20160 Mar 29 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.