9k views

### Kullback-Leibler divergence - interpretation [duplicate]

I have a question about the Kullback-Leibler divergence. Can someone explain why the "distance" between the blue density and the "red" density is smaller than the distance between the "green" curve ...
1k views

### Why is Kullback-Leilbler divergence a better metric for measuring distance between two probability distributions than squared error? [duplicate]

I know that KL-divergence is a metric that is more suitable when we want to measure the distance between numbers which a probability form. However, I am still confused what is the benefit of using KL-...
822 views

### textbook example of KL Divergence [duplicate]

I have read what KL Divergence is about: assess differences in probability distributions between two sets. I have also read, and digested, that it is emphatically not a true metric because of ...
443 views

### Where does the Kullback-Leibler come from? [duplicate]

Let $p(x)$ be some "true" distribution which we want to model using a simpler distribution $q(x)$. Why is the KL divergence $$KL(q||p)=\int q(x)\log{\frac{q(x)}{p(x)}}$$ a good way to represents the ...
616 views

### KL divergence vs Absolute Difference between two distributions? [duplicate]

Why should I use KL divergence over just giving the abs difference from two PDFs?
104 views

### Kullback-Leibler divergence with sample data likelihood [duplicate]

I'm trying to get my head around the KL divergence in the context of the sample likelihood under two competing hypotheses, one optimal $H_0$ and one suboptimal $H_1$. Roughly speaking, I want to see ...
21k views

### Why do we use Kullback-Leibler divergence rather than cross entropy in the t-SNE objective function?

In my mind, KL divergence from sample distribution to true distribution is simply the difference between cross entropy and entropy. Why do we use cross entropy to be the cost function in many machine ...
3k views

### Why should we use t errors instead of normal errors?

In this blog post by Andrew Gelman, there is the following passage: The Bayesian models of 50 years ago seem hopelessly simple (except, of course, for simple problems), and I expect the Bayesian ...
2k views

### Kullback-Leibler divergence WITHOUT information theory

After much trawling of Cross Validated, I still don't feel like I'm any closer to understanding KL divergence outside of the realm of information theory. It's rather odd as somebody with a Math ...
5k views

### What's the maximum value of Kullback-Leibler (KL) divergence

I am going to use KL divergence in my python code and I got this tutorial. On that tutorial, to implement KL divergence is quite simple. ...
9k views

Both of Cross-entropy and KL divergence are tools to measure the distance between two probability distribution. What is the difference? $$H(P,Q) = -\sum_x P(x)\log Q(x)$$ $$KL(P | Q) = \sum_{x} P(... 1answer 6k views ### Qualitively what is Cross Entropy This question gives a quantitative definition of cross entropy, in terms of it's formula. I'm looking for a more notional definition, wikipedia says: In information theory, the cross entropy ... 2answers 8k views ### Kullback-Leibler Divergence for two samples I tried to implement a numerical estimate of the Kullback-Leibler Divergence for two samples. To debug the implementation draw the samples from two normal distributions \mathcal N (0,1) and \... 2answers 5k views ### Intuition of the Bhattacharya Coefficient and the Bhattacharya distance? The Bhattacharyya distance is defined as D_B(p,q) = -\ln \left( BC(p,q) \right), where BC(p,q) = \sum_{x\in X} \sqrt{p(x) q(x)} for discrete variables and similarly for continuous random variables.... 1answer 6k views ### Intuitively, why is cross entropy a measure of distance of two probability distributions? For two discrete distributions p and q, cross entropy is defined as$$H(p,q)=-\sum_x p(x)\log q(x). I wonder why this would be an intuitive measure of distance between two probability ...

15 30 50 per page