9k views

Kullback-Leibler divergence - interpretation [duplicate]

I have a question about the Kullback-Leibler divergence. Can someone explain why the "distance" between the blue density and the "red" density is smaller than the distance between the "green" curve ...
1k views

Why is Kullback-Leilbler divergence a better metric for measuring distance between two probability distributions than squared error? [duplicate]

I know that KL-divergence is a metric that is more suitable when we want to measure the distance between numbers which a probability form. However, I am still confused what is the benefit of using KL-...
822 views

textbook example of KL Divergence [duplicate]

I have read what KL Divergence is about: assess differences in probability distributions between two sets. I have also read, and digested, that it is emphatically not a true metric because of ...
443 views

Where does the Kullback-Leibler come from? [duplicate]

Let $p(x)$ be some "true" distribution which we want to model using a simpler distribution $q(x)$. Why is the KL divergence $$KL(q||p)=\int q(x)\log{\frac{q(x)}{p(x)}}$$ a good way to represents the ...
616 views

KL divergence vs Absolute Difference between two distributions? [duplicate]

Why should I use KL divergence over just giving the abs difference from two PDFs?
104 views

Kullback-Leibler divergence with sample data likelihood [duplicate]

I'm trying to get my head around the KL divergence in the context of the sample likelihood under two competing hypotheses, one optimal $H_0$ and one suboptimal $H_1$. Roughly speaking, I want to see ...
21k views

Why do we use Kullback-Leibler divergence rather than cross entropy in the t-SNE objective function?

In my mind, KL divergence from sample distribution to true distribution is simply the difference between cross entropy and entropy. Why do we use cross entropy to be the cost function in many machine ...
3k views

Why should we use t errors instead of normal errors?

In this blog post by Andrew Gelman, there is the following passage: The Bayesian models of 50 years ago seem hopelessly simple (except, of course, for simple problems), and I expect the Bayesian ...
2k views

Kullback-Leibler divergence WITHOUT information theory

After much trawling of Cross Validated, I still don't feel like I'm any closer to understanding KL divergence outside of the realm of information theory. It's rather odd as somebody with a Math ...