10
$\begingroup$

I know that KL Divergence is not symmetric and it cannot be strictly considered as a metric. If so, why is it used when JS Divergence satisfies the required properties for a metric?

Are there scenarios where KL divergence can be used but not JS Divergence or vice-versa?

$\endgroup$
  • $\begingroup$ They are both used, only it depends on the context. When it's clear that it's necessary to have a strict metric, e.g. when clustering is done, then JS is a more preferable choice. On the other hand, in model selection the usage of AIC which is based on KL is widespread. Akaike weights have a nice interpretation for which JS either can't provide a counterpart or it has yet to become popular. $\endgroup$ – James Sep 29 '14 at 20:23
1
$\begingroup$

KL divergence has clear information theoretical interpretation and is well-known; but I am first time to hear that the symmetrization of KL divergence is called JS divergence. The reason that JS-divergence is not so often used is probably that it is less well-known and does not offer must-have properties.

$\endgroup$
1
$\begingroup$

I found a very mature answer on the Quora and just put it here for people who look for it here:

The Kullback-Leibler divergence has a few nice properties, one of them being that $𝐾𝐿[𝑞;𝑝]$ kind of abhors regions where $𝑞(𝑥)$ have non-null mass and $𝑝(𝑥)$ has null mass. This might look like a bug, but it’s actually a feature in certain situations.

If you’re trying to find approximations for a complex (intractable) distribution $𝑝(𝑥)$ by a (tractable) approximate distribution $𝑞(𝑥)$ you want to be absolutely sure that any 𝑥 that would be very improbable to be drawn from $𝑝(𝑥)$ would also be very improbable to be drawn from $𝑞(𝑥)$. That KL have this property is easily shown: there’s a $𝑞(𝑥)𝑙𝑜𝑔[𝑞(𝑥)/𝑝(𝑥)]$ in the integrand. When 𝑞(𝑥) is small but $𝑝(𝑥)$ is not, that’s ok. But when $𝑝(𝑥)$ is small, this grows very rapidly if $𝑞(𝑥)$ isn’t also small. So, if you’re choosing $𝑞(𝑥)$ to minimize $𝐾𝐿[𝑞;𝑝]$, it’s very improbable that $𝑞(𝑥)$ will assign a lot of mass on regions where $𝑝(𝑥)$ is near zero.

The Jensen-Shannon divergence don’t have this property. It is well behaved both when $𝑝(𝑥)$ and $𝑞(𝑥)$ are small. This means that it won’t penalize as much a distribution $𝑞(𝑥)$ from which you can sample values that are impossible in $𝑝(𝑥)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.