17
$\begingroup$

I know that KL Divergence is not symmetric and it cannot be strictly considered as a metric. If so, why is it used when JS Divergence satisfies the required properties for a metric?

Are there scenarios where KL divergence can be used but not JS Divergence or vice-versa?

$\endgroup$
1
  • $\begingroup$ They are both used, only it depends on the context. When it's clear that it's necessary to have a strict metric, e.g. when clustering is done, then JS is a more preferable choice. On the other hand, in model selection the usage of AIC which is based on KL is widespread. Akaike weights have a nice interpretation for which JS either can't provide a counterpart or it has yet to become popular. $\endgroup$ – James Sep 29 '14 at 20:23
9
$\begingroup$

I found a very mature answer on the Quora and just put it here for people who look for it here:

The Kullback-Leibler divergence has a few nice properties, one of them being that $𝐾𝐿[π‘ž;𝑝]$ kind of abhors regions where $π‘ž(π‘₯)$ have non-null mass and $𝑝(π‘₯)$ has null mass. This might look like a bug, but it’s actually a feature in certain situations.

If you’re trying to find approximations for a complex (intractable) distribution $𝑝(π‘₯)$ by a (tractable) approximate distribution $π‘ž(π‘₯)$ you want to be absolutely sure that any π‘₯ that would be very improbable to be drawn from $𝑝(π‘₯)$ would also be very improbable to be drawn from $π‘ž(π‘₯)$. That KL have this property is easily shown: there’s a $π‘ž(π‘₯)π‘™π‘œπ‘”[π‘ž(π‘₯)/𝑝(π‘₯)]$ in the integrand. When π‘ž(π‘₯) is small but $𝑝(π‘₯)$ is not, that’s ok. But when $𝑝(π‘₯)$ is small, this grows very rapidly if $π‘ž(π‘₯)$ isn’t also small. So, if you’re choosing $π‘ž(π‘₯)$ to minimize $𝐾𝐿[π‘ž;𝑝]$, it’s very improbable that $π‘ž(π‘₯)$ will assign a lot of mass on regions where $𝑝(π‘₯)$ is near zero.

The Jensen-Shannon divergence don’t have this property. It is well behaved both when $𝑝(π‘₯)$ and $π‘ž(π‘₯)$ are small. This means that it won’t penalize as much a distribution $π‘ž(π‘₯)$ from which you can sample values that are impossible in $𝑝(π‘₯)$.

$\endgroup$
1
$\begingroup$

KL divergence has clear information theoretical interpretation and is well-known; but I am first time to hear that the symmetrization of KL divergence is called JS divergence. The reason that JS-divergence is not so often used is probably that it is less well-known and does not offer must-have properties.

$\endgroup$

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.