# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

287 questions
Filter by
Sorted by
Tagged with
14 views

### How to measure the “variability” across a set of many (>>2) probability distributions?

Given a set of many of discrete probability distributions, is there a way I can efficiently calculate a metric that quantifies how different the entire set of these probability distributions are to ...
55 views

### KL-Diverence of Q(z|X) and P(z) in Variational Autoencoder (VAE)

I aim to understand how $D_{KL}[Q(z | X) || P(z)]$ can be converted to $\frac{1}{2} \sum_{k} (\Sigma(X) + \mu^{2}(X) - 1 - \log \Sigma(X))$, where $k$ is the dimension of the Gaussian distribution. ...
236 views

### Kullback-Leibler divergence with huge values? [closed]

I was playing with the KL divergence. My simple example is to calculate the divergence between two 2-dimensional normal distribution using PyTorch. The code for doing it is just below. I get an ...
22 views

### Confusion about maximising vs minimising (relative) entropy terminology and methods

I'm studying rules of inference for updating from a prior probability distribution to a posterior. One method for doing this is by maximising entropy, subject to constraints. I'm reading papers like ...
12 views

### Are these inequalites correct? (between variational distance and bhattacharyya distance and KL divergence)

I have some proof in terms of the definition explained
149 views

### Can we apply KL divergence to the probability distributions on different domains?

When I was reading the original paper of t-SNE, I had an question whether or not we can apply KL divergence to the discrete probability distributions on different domains. In the paper, they measure ...
200 views

### Difference of notation between cross entropy and joint entropy

Although it is clear to me, how the two concepts differs, it has been difficult for me to find a notation that would make it clear, to which type of entropy we refer. From wikipedia, we can see that ...
80 views

### Kullback-Leibler Divergence as the class separation measure

I am trying to use KL divergence as the separation measure between the classes. I have the positive, negative samples for 2 distributions and want to adjust the algorithm parameters to get the best ...
21 views

### Solution of $D_{KL}(q_\phi(z)||p_\theta(z))$ of Gaussian case

Following is from the original paper of concept of VAE(variational autoencoder) by Kingma,Welling 2014 B. Solution of $D_{KL}(q_\phi(z)||p_\theta(z))$ of Gaussian case The variational lower bound (...
249 views

### KL Divergence with different domains

I want to calculate KL Divergence between a normal and an exponential r.v. i.e. $$D(P||Q) = ?\\ \;\; P=N(\mu,\sigma), \;\; Q=exp(\lambda)$$ My problem is that in this case the domains of the ...
505 views

### Correcting Kullback-Leibler divergence for size of datasets

We have the following implementation of KLD: ...
31 views

### Derivation of ELBO upon the Existence of Conditional Latent Variable Model

I am reading the recently published paper from DeepMind, "Neural Scene Representation and Rendering" and especially its "Supplementary Materials". Following is the page 1 and it's pretty hard for ...
201 views

### Finding the value of KL divergence to determine whether one distribution is distrinct from another?

Given the KL divergence value between 2 distributions, how should someone use this to determine whether the value is significant for the distributions $P$ and $Q$ to be different? One method I can ...
51 views

### Distance between angle distributions

I want to quantify the complexity of the street network of different cities. For each city I have the angle distribution of its streets. My hypothesis that the more complex the street network, the ...
168 views

### Why minimize the KL divergence from data to the model instead of KL divergence of model to data?

When minimizing the KL divergence in machine learning, why the expression of KL is from data to model, instead of from model to data ?
1k views

### KL divergence between which distributions could be infinity

I know that KL divergence measures difference between two probability distributions. My doubt is for which of the distributions it could become Infinity, putting it in another way, ...
3k views

### What is the difference Cross-entropy and KL divergence?

Both of Cross-entropy and KL divergence are tools to measure the distance between two probability distribution. What is the difference? $$H(P,Q) = -\sum_x P(x)\log Q(x)$$  KL(P | Q) = \sum_{x} P(...
309 views

### Rate of converge of KL-divergence to posterior

Suppose you have samples from some distribution P. You have a prior distribution Q, which represents your estimate of P, and assume for now that it's parameterized the same way as P. Upon observing ...
120 views

### Difference between Empirical distribution and Bernoulli distribution

I've been studying binary cross entropy error for binary classification weight optimization. From my knowledge, Cross entropy itself quantifies divergence between two probability distributions with ...
349 views

### Why Kullback-Leibler in Stochastic Neighbor Embedding

Stochastic Neighbor Embedding (and t-SNE) relies on Kullback-Leibler divergence between the point distributions in the original and the low-dimensional space. Why? Why not any other dissimilarity ...
39 views

### Choosing an appropriate statistical distance which punishes entropy

Problem Description: I have with me experimental statistics of a system and I wish to fit a theoretical model so that the computed statistics on the model fit the experimental ones. I am using an RBM ...
3k views

### What's the maximum value of Kullback-Leibler (KL) divergence

I am going to use KL divergence in my python code and I got this tutorial. On that tutorial, to implement KL divergence is quite simple. ...
88 views

### Comparing distribution A to B and C

I have three discrete probability distributions, A, B and C. They are all measuring P(X) under different circumstances. I suspect that A is more similar to B than it is to C. I know that I can compare ...
128 views

### Kullback–Leibler divergence when one measure is a sum of diracs

In the book "Deep Learning" of Goodfellow, Bengio and Courville, section 5.5 of maximum likelihood estimation they explain a relation between the maximization of likelihood and minimization of the K-L ...
57 views

### What is the intuition regarding the different viewpoints of the Kullback-Leibler Divergence and the Kolmogorov-Smirnov Statistic? [duplicate]

In trying to understand the Kullback-Leibler Divergence I conceive it as a metric that if minimized would make the Approximation PDF Q as close as possible to the True PDF P either in the ...
209 views

### Calculating Jensen-Shannon and Kullback-Leibler for density splines

I need to find a way to calculate the JS distance (and, by necessity, the KL divergence) between continuous, non-normal (generating distribution unknown) distributions. I have many classes, and ...
129 views

### Does this bounded, continuous probability distribution have a name?

Does this bounded, continuous probability distribution over $x$ have a name? $P(x|y) \propto \big(\frac{y}{x}\big)^x\big(\frac{1-y}{1-x}\big)^{(1-x)}$ for $x, y \in (0,1)$. This comes about by ...
223 views

### If the AIC is an estimate of Kullback-Leibler Divergence, then why can AIC be negative when KL divergence is always positive?

I have read many times that the AIC serves as an estimate of the KL divergence, and I know that AIC can be a negative value (and have seen that myself). Yet, the KL divergence must always be positive. ...
408 views

176 views

### Connection between log predictive density and Kullback-Leibler information measure

I have read in Bayesian Data Analysis by Andrew Gelman that log predictive density can be used to compare Bayesian models due to its connection to the Kullback-Leibler information. The log ...
282 views

### Calculate the Kullback-Leibler Divergence for these 2 Gamma distributions

I have 2 models $P \sim Ga(115,1329.914) \\ Q \sim Ga(140,650.6775)$ and I'm looking to calculate the K-L divergence of these 2. $D_{KL}(P||Q) = \int_\infty ^\infty p(x) log \frac{p(x)}{q(x)}\,dx$...
357 views

### How to picture EM algorithm and KL-divergence geometrically?

In reading up on the Expectation-Mmaximization algorithm on Wikipedia, I read this short and intriguing line, under the subheading "Geometric Intuition": In information geometry, the E step and the ...
8k views

### Why KL divergence is non-negative?

Why is KL divergence non-negative? From the perspective of information theory, I have such an intuitive understanding: Say there are two ensembles $A$ and $B$ which are composed of the same set of ...
500 views

### Upper bound on KL divergence

Is there a maximum (unique?) to the KL divergence between discrete distributions p & q, with the restriction that q is a proper probability distribution? I know KL is unbounded from above when q ...
1k views

### Why don't we use a symmetric cross-entropy loss?

Machine learning classifiers often use the cross-entropy $\mathbb{H}[p,q]$, where $p$ is the true distribution (often a delta) and $q$ is the predicted distribution over classes (or can at least be ...
196 views

### Distance or divergence for ordinal distribution

Measures like KL divergence can be symmetrized (into JS divergence). Bhattacharyya distance serves a similar function. Either is well-suited to both continuous distributions and discrete (e.g. ...
54 views

### need help in understanding a research paper… specifically related to KL divergence

In this paper https://dl.acm.org/citation.cfm?id=2002654 specifically section 2.4. I understood that Review rating is modelled as random variable with guassian distribution with mean as wt * or. I ...
306 views

### Comparing ELBO of a VAE for different samples

I am lacking of an interpretation of the evidence lower bound (ELBO), when comparing two different samples $x_1, x_2 \sim X$. Writing the marginal log-likelihood as the sum of lower variational bound ...
236 views

### How to justify a KL divergence when a distribution contains continuous and discrete components

Like in contamination models, some distributions have discrete component. e.g. $p(x) := (1 - \epsilon) q(x) + \epsilon \delta_{x_0}(x)$ In these distributions, is there a way to justify a definition ...
161 views

### KL divergence and anomaly detection

stats newbie here. I have a dataset that is collected weekly. In order to make sure the data set gathered this week conform to past observations, I'm using KL divergence to compute how similar the ...
54 views

### Can we state that If KL-Divergence(P||Q) < H(P) then Q is “informative” of P and not otherwise?

From what I've read the KL-Divergence between $P||Q$ is the extra amount of "bits" you need to describe $P$ if you are encoding it with $Q$.(Analysis of Kullback-Leibler divergence). I want to know ...
934 views

### Interpretation of Radon-Nikodym derivative between probability measures?

I have seen at some points the use of the Radon-Nikodym derivative of one probability measure with respect to another, most notably in the Kullback-Leibler divergence, where it is the derivative of ...
50 views

### How to classify images with the same pattern, differing in color?

In order to increase your knowledge in the field of computer vision, I would like to create an application that will classify photos with the same pattern but with different colors. Is there any ...
99 views

### Questions about Mean-field variational inference

I am very new to this variational inference concept. I couldn't find any clear sources. I have two questions related to each other. Let's consider a very simple probabilistic model with a 2-D latent ...
3k views

### Can KL-Divergence ever be greater than 1?

I've been working on building some test statistics based on the KL-Divergence, \begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation} And I ended up with a ...
I have a problem with solving the following question. Let $\mathcal{P} = \{\mathbb{P}_\theta : \theta \in \Theta\}$ be a statistical family of discrete distributions with state space $\mathcal{X}$ ...
Bhattacharyya distance between two distributions $p$ and $q$ is defined as $D_B(p,q)=-\log(\int\sqrt{p(x)q(x)})dx$, The KL-divergence is defined as $D_{kl}(p||q)=\int p(x)\log(\frac{p(x)}{q(x)})dx$. ...