125
votes
Accepted
Why do we use Kullback-Leibler divergence rather than cross entropy in the t-SNE objective function?
KL divergence is a natural way to measure the difference between two probability distributions. The entropy $H(p)$ of a distribution $p$ gives the minimum possible number of bits per message that ...
- 33.1k
117
votes
Accepted
What is the difference between Cross-entropy and KL divergence?
You will need some conditions to claim the equivalence between minimizing cross entropy and minimizing KL divergence. I will put your question under the context of classification problems using cross ...
- 1,848
114
votes
Accepted
What is the advantages of Wasserstein metric compared to Kullback-Leibler divergence?
When considering the advantages of Wasserstein metric compared to KL divergence, then the most obvious one is that W is a metric whereas KL divergence is not, since KL is not symmetric (i.e. $D_{KL}(P|...
- 1,316
78
votes
Accepted
Why KL divergence is non-negative?
Proof 1:
First note that $\ln a \leq a-1$ for all $a \gt 0$.
We will now show that $-D_{KL}(p||q) \leq 0$ which means that $D_{KL}(p||q) \geq 0$
\begin{align}
-D(p||q)&=-\sum_x p(x)\ln \frac{p(...
- 1,495
52
votes
Accepted
Deriving the KL divergence loss for VAEs
The encoder distribution is $q(z|x)=\mathcal{N}(z|\mu(x),\Sigma(x))$ where $\Sigma=\text{diag}(\sigma_1^2,\ldots,\sigma^2_n)$.
The latent prior is given by $p(z)=\mathcal{N}(0,I)$.
Both are ...
- 2,244
39
votes
What is the advantages of Wasserstein metric compared to Kullback-Leibler divergence?
Wasserstein metric most commonly appears in optimal transport problems where the goal is to move things from a given configuration to a desired configuration in the minimum cost or minimum distance. ...
- 4,299
37
votes
What is the difference between Cross-entropy and KL divergence?
I suppose it is because the models usually work with the samples packed in mini-batches. For KL divergence and Cross-Entropy, their relation can be written as
$$H(p, q) = D_{KL}(p, q)+H(p) = -\sum_i{...
- 479
28
votes
Accepted
Why are there extra terms $-p_i+q_i$ in SciPy's implementation of Kullback-Leibler divergence?
The other answer tells us why we don't usually see the $-p_i+q_i$ term: $p$ and $q$ are usually residents of the simplex and so sum to one, so this leads to $\sum - [p_i - q_i] = \sum - p_i + \sum q_i ...
- 4,650
27
votes
Kullback-Leibler divergence WITHOUT information theory
There is a purely statistical approach to Kullback-Leibler divergence: take a sample $X_1,\ldots,X_n$ iid from an unknown distribution $p^\star$ and consider the potential fit by a family of ...
- 107k
25
votes
What's the maximum value of Kullback-Leibler (KL) divergence
Or even with the same support, when one distribution has a much fatter tail than the other. Take
$$KL(P\vert\vert Q) = \int p(x)\log\left(\frac{p(x)}{q(x)}\right) \,\text{d}x$$
when
$$p(x)=\overbrace{\...
- 107k
24
votes
Accepted
How should I intuitively understand the KL divergence loss in variational autoencoders?
The KL divergence tells us how well the probability distribution Q approximates the probability distribution P by calculating the cross-entropy minus the entropy. Intuitively, you can think of that as ...
- 428
22
votes
Accepted
Can KL-Divergence ever be greater than 1?
The Kullback-Leibler divergence is unbounded. Indeed, since there is no lower bound on the $q(i)$'s, there is no upper bound on the $p(i)/q(i)$'s. For instance, the Kullback-Leibler divergence ...
- 107k
22
votes
Why are there extra terms $-p_i+q_i$ in SciPy's implementation of Kullback-Leibler divergence?
The referenced book has a free pdf on Boyd's site: https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf
On page 90, formula 3.17 gives this definition. I suspect the reason for the added terms is ...
- 4,896
21
votes
Connection between Fisher metric and the relative entropy
Proof for usual (non-symmetric) KL divergence
Zen's answer uses the symmetrized KL divergence, but the result holds for the usual form as well, since it becomes symmetric for infinitesimally close ...
- 311
19
votes
Accepted
Why is Kullback-Leilbler divergence a better metric for measuring distance between two probability distributions than squared error?
Overview:
KL-Divergence is derived from the Shannon entropy.
The Shannon entropy is the amount of information contained in a signal X with distribution $\mathrm{P}(X)$.
The cross entropy is the ...
- 12.1k
19
votes
Accepted
Are Mutual Information and Kullback–Leibler divergence equivalent?
Mutual information is not a metric. A metric $d$ satisfies the identity of indiscernibles: $d(x, y) = 0$ if and only if $x = y$. This is not true of mutual information, which behaves in the opposite ...
- 33.1k
18
votes
Accepted
Kullback-Leibler Divergence for two samples
The Kullback-Leibler divergence is defined as
$$
\DeclareMathOperator{\KL}{KL}
\KL(P || Q) = \int_{-\infty}^\infty p(x) \log \frac{p(x)}{q(x)} \; dx
$$
so to calculate (estimate) this from ...
- 81.7k
18
votes
Jensen Shannon Divergence vs Kullback-Leibler Divergence?
I found a very mature answer on the Quora and just put it here for people who look for it here:
The Kullback-Leibler divergence has a few nice properties, one of them
being that $𝐾𝐿[𝑞;𝑝]$ ...
- 526
17
votes
What's the maximum value of Kullback-Leibler (KL) divergence
For distributions which do not have the same support, KL divergence is not bounded. Look at the definition:
$$KL(P\vert\vert Q) = \int_{-\infty}^{\infty} p(x)\ln\left(\frac{p(x)}{q(x)}\right) dx$$
...
- 694
17
votes
KL divergence between which distributions could be infinity
What happens to $D_{KL}(p \parallel q)$ when $p(x)$ and/or $q(x)$ is zero? In a strict sense, the log of zero is undefined because there's no value of $x$ such that $e^x = 0$. But, the definition of ...
- 33.1k
15
votes
Distance between two Gaussian mixtures to evaluate cluster solutions
Suppose we have two Gaussian mixtures in $\mathbb R^d$:$\DeclareMathOperator{\N}{\mathcal N} \newcommand{\ud}{\mathrm{d}} \DeclareMathOperator{\E}{\mathbb E} \DeclareMathOperator{\MMD}{\mathrm{MMD}}$
$...
- 25.2k
15
votes
Is it possible to apply KL divergence between discrete and continuous distribution?
Yes, the KL divergence between continuous and discrete random variables is well defined. If $P$ and $Q$ are distributions on some space $\mathbb{X}$, then both $P$ and $Q$ have densities $f$, $g$ with ...
- 819
15
votes
Accepted
Why don't we use a symmetric cross-entropy loss?
Consider a classification context like you mentioned, where $q(y \mid x)$ is the model distribution over classes, given input $x$. $p(y \mid x)$ is the 'true' distribution, defined as a delta function ...
- 33.1k
14
votes
Interpretation of Radon-Nikodym derivative between probability measures?
First, we don't need probability measures, just $\sigma$-finiteness. So let $\mathcal M = (\Omega, \mathscr F)$ be a measurable space and let $\mu$ and $\nu$ be $\sigma$-finite measures on $\mathcal M$...
- 20.6k
11
votes
Accepted
What is the meaning of || (double vertical bar) in this KL divergence equation?
My understanding is that the double bar emphasises that the order of the arguments matters. The reminder is perhaps helpful because KL is used much like a distance, but it's not symmetric, so it's not ...
- 4,326
11
votes
Estimate the Kullback–Leibler (KL) divergence with Monte Carlo
I assume you can evaluate $f$ and $g$ up to a normalizing constant. Denote $f(x) = f_u(x)/c_f$ and $g(x) = g_u(x)/c_g$.
A consistent estimator that may be used is
$$
\widehat{D_{KL}}(f || g) = \left[...
- 21.3k
11
votes
Accepted
Kullback–Leibler divergence
I just wonder why he wants to measure the similarity between the
distributions $p(x|\theta)$ and $p(x|\theta_0)$.
You're kind of asking the wrong question. If we're in a setting where we're using ...
- 1,618
10
votes
Kullback-Leibler divergence WITHOUT information theory
Here is a statistical interpretation of the Kullback-Leibler divergence, loosely taken from I.J. Good (Weight of evidence: A brief survey, Bayesian Statistics 2, 1985).
The weight of evidence.
...
- 819
10
votes
Accepted
Cross entropy vs KL divergence: What's minimized directly in practice?
Let $q$ be the density of your true data-generating process and $f_\theta$ be your model-density.
Then $$KL(q||f_\theta) = \int q(x) log\left(\frac{q(x)}{f_\theta(x)}\right)dx = -\int q(x) \log(f_\...
- 3,104
9
votes
Accepted
Does the Jensen-Shannon divergence maximise likelihood?
First, it is important to clarify a few things.
The KL divergence is a dissimilarity between two distributions, so it cannot maximize the likelihood, which is a function of a single distribution.
...
- 14.8k
Only top scored, non community-wiki answers of a minimum length are eligible