Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
76
votes
2answers
71k views

KL divergence between two univariate Gaussians

I need to determine the KL-divergence between two Gaussians. I am comparing my results to these, but I can't reproduce their result. My result is obviously wrong, because the KL is not 0 for KL(p, p). ...
45
votes
5answers
13k views

Intuition on the Kullback-Leibler (KL) Divergence

I have learned about the intuition behind the KL Divergence as how much a model distribution function differs from the theoretical/true distribution of the data. The source I am reading goes on to say ...
42
votes
1answer
37k views

KL divergence between two multivariate Gaussians

I'm having trouble deriving the KL divergence formula assuming two multivariate normal distributions. I've done the univariate case fairly easily. However, it's been quite a while since I took math ...
38
votes
1answer
18k views

Why do we use Kullback-Leibler divergence rather than cross entropy in the t-SNE objective function?

In my mind, KL divergence from sample distribution to true distribution is simply the difference between cross entropy and entropy. Why do we use cross entropy to be the cost function in many machine ...
36
votes
2answers
7k views

Kullback–Leibler vs Kolmogorov-Smirnov distance

I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures. However, both are used to measure the distance between distributions. Is there a ...
30
votes
2answers
9k views

Differences between Bhattacharyya distance and KL divergence

I'm looking for an intuitive explanation for the following questions: In statistics and information theory, what's the difference between Bhattacharyya distance and KL divergence, as measures of the ...
28
votes
4answers
5k views

An adaptation of the Kullback-Leibler distance?

Look at this picture: If we draw a sample from the red density then some values are expected to be less than 0.25 whereas it is impossible to generate such a sample from the blue distribution. As a ...
28
votes
4answers
22k views

Measures of similarity or distance between two covariance matrices

Are there any measures of similarity or distance between two symmetric covariance matrices (both having the same dimensions)? I am thinking here of analogues to KL divergence of two probability ...
23
votes
3answers
1k views

Kullback-Leibler divergence WITHOUT information theory

After much trawling of Cross Validated, I still don't feel like I'm any closer to understanding KL divergence outside of the realm of information theory. It's rather odd as somebody with a Math ...
21
votes
2answers
6k views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...
20
votes
3answers
6k views

Connection between Fisher metric and the relative entropy

Can someone prove the following connection between Fisher information metric and the relative entropy (or KL divergence) in a purely mathematical rigorous way? $$D( p(\cdot , a+da) \parallel p(\cdot,...
20
votes
2answers
4k views

What is the advantages of Wasserstein metric compared to Kullback-Leibler divergence?

What is the practical difference between Wasserstein metric and Kullback-Leibler divergence? Wasserstein metric is also referred to as Earth mover's distance. From Wikipedia: Wasserstein (or ...
17
votes
2answers
9k views

Kullback-Leibler divergence - interpretation [duplicate]

I have a question about the Kullback-Leibler divergence. Can someone explain why the "distance" between the blue density and the "red" density is smaller than the distance between the "green" curve ...
14
votes
3answers
11k views

Calculate the Kullback-Leibler Divergence in practice?

I am using KL Divergence as a measure of dissimilarity between 2 $p.m.f.$ $P$ and $Q$. $$D_{KL}(P||Q) = \sum_{i=1}^N \ln \left( \frac{P_i}{Q_i} \right) P_i$$ $$=-\sum P(X_i)ln\left(Q(X_i)\right) + \...
14
votes
4answers
8k views

Questions about KL divergence?

I am comparing two distributions with KL divergence which returns me a non-standardized number that, according to what I read about this measure, is the amount of information that is required to ...
14
votes
2answers
4k views

Kullback–Leibler divergence between two gamma distributions

Choosing to parameterize the gamma distribution $\Gamma(b,c)$ by the pdf $g(x;b,c) = \frac{1}{\Gamma(c)}\frac{x^{c-1}}{b^c}e^{-x/b}$ The Kullback-Leibler divergence between $\Gamma(b_q,c_q)$ and $\...
13
votes
3answers
13k views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
12
votes
3answers
3k views

What's the maximum value of Kullback-Leibler (KL) divergence

I am going to use KL divergence in my python code and I got this tutorial. On that tutorial, to implement KL divergence is quite simple. ...
12
votes
1answer
8k views

Why KL divergence is non-negative?

Why is KL divergence non-negative? From the perspective of information theory, I have such an intuitive understanding: Say there are two ensembles $A$ and $B$ which are composed of the same set of ...
12
votes
1answer
258 views

Special probability distribution

If $p(x)$ is a probability distribution with non-zero values on $[0,+\infty)$, for what type(s) of $p(x)$ does there exist a constant $c\gt 0$ such that $\int_0^{\infty}p(x)\log{\frac{ p(x)}{(1+\...
11
votes
1answer
934 views

Interpretation of Radon-Nikodym derivative between probability measures?

I have seen at some points the use of the Radon-Nikodym derivative of one probability measure with respect to another, most notably in the Kullback-Leibler divergence, where it is the derivative of ...
10
votes
2answers
20k views

How to calculate Kullback-Leibler divergence/distance?

I have three data sets X, Y and Z. Each data set defines the frequency of an event occurring. For example: Data Set X: E1:4, E2:0, E3:10, E4:5, E5:0, E6:0 and so on.. Data Set Y: E1:2, E2:3, E3:7, ...
10
votes
2answers
5k views

Hypothesis testing and total variation distance vs. Kullback-Leibler divergence

In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...
10
votes
3answers
2k views

Distance between two Gaussian mixtures to evaluate cluster solutions

I'm running a quick simulation to compare different clustering methods, and currently hit a snag trying to evaluate the cluster solutions. I know of various validation metrics (many found in cluster....
9
votes
3answers
5k views

Is it possible to apply KL divergence between discrete and continuous distribution?

I am not a mathematician. I have searched the internet about KL Divergence. What I learned is the the KL divergence measures the information lost when we approximate distribution of a model with ...
9
votes
1answer
161 views

Why does the Bayesian posterior concentrate around the minimiser of KL divergence?

Consider the Bayesian posterior $\theta\mid X$. Asymptotically, its maximum occurs at the MLE estimate $\hat \theta$, which just maximizes the likelihood $\operatorname{argmin}_\theta\, f_\theta(X)$. ...
9
votes
2answers
5k views

Population stability index - division by zero

Population stability index quantifies the change of a distribution of a variable by comparing data samples in two time periods. It is very commonly used to measure shifts in scores. It is calculated ...
9
votes
2answers
3k views

Jensen Shannon Divergence vs Kullback-Leibler Divergence?

I know that KL Divergence is not symmetric and it cannot be strictly considered as a metric. If so, why is it used when JS Divergence satisfies the required properties for a metric? Are there ...
8
votes
2answers
2k views

Estimate the Kullback Leibler (KL) divergence with monte carlo

I want to estimate the KL divergence between two continuous distributions f and g. However, I can't write down the density for either f or g. I can sample from both f and g via some method (for ...
8
votes
2answers
583 views

Why is there a E in the name EM algorithm?

I understand where the E step happens in the algorithm (as explicated in the math section below). In my mind, the key ingenuity of the algorithm is the use of the Jensen's inequality to create a lower ...
8
votes
2answers
6k views

Kullback-Leibler Divergence

I tried to implement a numerical estimate of the Kullback-Leibler Divergence for two samples. To debug the implementation draw the samples from two normal distributions $\mathcal N (0,1)$ and $\...
8
votes
2answers
2k views

KL Loss with a unit Gaussian

I've been implementing a VAE and I've noticed two different implementations online of the simplified univariate gaussian KL divergence. The original divergence as per here is $$ KL_{loss}=\log(\frac{\...
7
votes
3answers
5k views

How to compute the Kullback-Leibler divergence when the PMF contains 0s?

I have the following timeseries obtained using the data posted below. For a sliding window size of 10, I am trying to compute the KL-divergence between the PMF of values within the current sliding ...
7
votes
3answers
480 views

Choose probability distribution to maximize evaluation function (for CDC flu forecasting contest)

Suppose you have a discrete random variable $X$ with probability mass function $p(x) = P(X=x)$ on the support $0,\ldots,n$. What function $q(x)\ge 0$ such that $\sum_{x=0}^n q(x) = 1$ maximizes $$ E(\...
7
votes
1answer
2k views

KL divergence and expectations

I am trying to understand the explanation of the KL divergence per below. It refers, as i understand it, to an expectation in the second term. "Approximating the expectation over q in this term". ...
6
votes
1answer
15k views

Kullback-Leibler divergence: negative values? [duplicate]

Wikipedia - KL properties says that KL can never be negative. But e.g. for texts where the probabilities are very small I somehow get negative values? E.g. Collection A: - word count: 321 doc count:...
6
votes
1answer
2k views

Kullback-Leibler divergence

Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal $p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler ...
6
votes
2answers
4k views

Use of KL Divergence in practice

It's not symmetric, so it can't really be used as a distance metric. I suppose given two known distributions p(x) and q(x), if one found another distribution z(x) but knew it came from either p or q,...
6
votes
2answers
1k views

Why KL-Divergence uses “ln” in its formula?

I notice in KL-Divergence formula a $ln$ function is used: $${D_{KL}}(P||Q) = \sum\limits_i {P(i)} \ln \frac{{P(i)}}{{Q(i)}},$$ where $i$ is a point and $P(i)$ the true discrete probability ...
6
votes
1answer
4k views

Estimate the Kullback-Leibler divergence

I would like to be sure I am able to compute the KL divergence based on a sample. Assume the data come from a Gamma distribution with shape=1/.85 and scale=.85. set.seed(937) theta <- ....
6
votes
1answer
341 views

How do I determine how well a dataset approximates a distribution?

Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another. I know that KL-divergence is a ...
6
votes
2answers
3k views

What is the difference Cross-entropy and KL divergence?

Both of Cross-entropy and KL divergence are tools to measure the distance between two probability distribution. What is the difference? $$ H(P,Q) = -\sum_x P(x)\log Q(x) $$ $$ KL(P | Q) = \sum_{x} P(...
6
votes
2answers
639 views

textbook example of KL Divergence [duplicate]

I have read what KL Divergence is about: assess differences in probability distributions between two sets. I have also read, and digested, that it is emphatically not a true metric because of ...
6
votes
1answer
630 views

KL divergence between a gamma distribution and a lognormal distribution?

Is there a closed-form formula for the following KL divergence? $D_{KL}(X,Y)$ where $X \sim \mathrm{Gamma}(k,\theta)$ and $Y \sim \mathrm{LogNormal}(\mu,\sigma^2)$
6
votes
1answer
3k views

What is the intuition behind the Population Stability Index?

The "Population Stability Index" for two distributions $P$ and $Q$ is defined as the Symmetrised Kullback-Leibler divergence: $$ \mathrm{PSI}(P,Q) = D_{KL}(P||Q) + D_{KL}(Q||P) = \sum_i(P_i-Q_i)\log\...
6
votes
1answer
126 views

$\min D_\textrm{KL}(p(x_1,\dots,x_n) \mid\mid q_1(x_1)\cdots q_n(x_n))$ gives the marginals of $p(x_1,\dots,x_n)$?

Prove or disprove: Let $p(x_1,\dots,x_n)$ be a given probability distribution over the $n$ variables $x_1, \dots,x_n$. The univariate probability distributions $q(x_1),\dots,q(x_n)$ that minimize the ...
6
votes
1answer
375 views

Multivariate time series model evaluation with conditional moments

Consider multivariate time series models that estimate potentially time-varying conditional means, variances, and correlations (one type of model might be a VAR(p)+Garch(1,1)+DCC Gaussian Copula model)...
6
votes
1answer
948 views

Kullback–Leibler divergence between two Wishart distributions

The result is shown in: [1] W.D. Penny, KL-Divergences of Normal, Gamma, Dirichlet, and Wishart densities, Available at: www.fil.ion.ucl.ac.uk/~wpenny/publications/densities.ps But could anyone help ...
5
votes
3answers
4k views

Is the square root of the symmetric Kullback-Leibler divergence a metric?

It is well known that the square root of the Jensen-Shannon divergence is a true metric, but how about the square root of symmetric KL: D(P||Q)+D(Q||P)? I have reasons to believe that it also is a ...
5
votes
1answer
3k views

Can KL-Divergence ever be greater than 1?

I've been working on building some test statistics based on the KL-Divergence, \begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation} And I ended up with a ...