9k views

Kullback-Leibler divergence - interpretation [duplicate]

I have a question about the Kullback-Leibler divergence. Can someone explain why the "distance" between the blue density and the "red" density is smaller than the distance between the "green" curve ...
634 views

textbook example of KL Divergence [duplicate]

I have read what KL Divergence is about: assess differences in probability distributions between two sets. I have also read, and digested, that it is emphatically not a true metric because of ...
401 views

Where does the Kullback-Leibler come from? [duplicate]

Let $p(x)$ be some "true" distribution which we want to model using a simpler distribution $q(x)$. Why is the KL divergence $$KL(q||p)=\int q(x)\log{\frac{q(x)}{p(x)}}$$ a good way to represents the ...
450 views

KL divergence vs Absolute Difference between two distributions? [duplicate]

Why should I use KL divergence over just giving the abs difference from two PDFs?
86 views

Kullback-Leibler divergence with sample data likelihood [duplicate]

I'm trying to get my head around the KL divergence in the context of the sample likelihood under two competing hypotheses, one optimal $H_0$ and one suboptimal $H_1$. Roughly speaking, I want to see ...
17k views

Why do we use Kullback-Leibler divergence rather than cross entropy in the t-SNE objective function?

In my mind, KL divergence from sample distribution to true distribution is simply the difference between cross entropy and entropy. Why do we use cross entropy to be the cost function in many machine ...
3k views

Why should we use t errors instead of normal errors?

In this blog post by Andrew Gelman, there is the following passage: The Bayesian models of 50 years ago seem hopelessly simple (except, of course, for simple problems), and I expect the Bayesian ...
1k views

Kullback-Leibler divergence WITHOUT information theory

After much trawling of Cross Validated, I still don't feel like I'm any closer to understanding KL divergence outside of the realm of information theory. It's rather odd as somebody with a Math ...
3k views

What's the maximum value of Kullback-Leibler (KL) divergence

I am going to use KL divergence in my python code and I got this tutorial. On that tutorial, to implement KL divergence is quite simple. ...