Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

2
votes
1answer
36 views

On a mistake computing the Kullback Liebler Information Criterion

THE FRAMEWORK: Let $X_1$ be an observation from a normal random variable with mean zero and variance $\sigma^2$ and lets call the PDF $f(x)$. I want to minimize the Kullback Liebler Information ...
0
votes
0answers
26 views

Bounding the KL divergence of a non-invertible transform of distributions

Let $p$ and $q$ be the distributions of random variables $x_1$ and $x_2$, and consider $p'$ and $q'$ to be the distributions of $g(x_1)$ and $g(x_2)$. For an invertible function $g$, it's true that ...
0
votes
0answers
6 views

What should the form of error be on CrossEntropy or KL-divergence loss function across samples of distributions?

Suppose your model produces (discrete) probability distributions and you have some truth distributions you want to compare to. For each sample $i$, you can compute the loss as the KL divergence or ...
0
votes
0answers
28 views

When do these two definitions of KL-divergence match?

Suppose $P$ and $Q$ are two distributions on a space ${\cal H}$ (could be a subset of an infinite dimensional function space) with p.d.fs denoted by the same letter then one can define the $KL$ ...
0
votes
1answer
33 views
0
votes
1answer
52 views

Total Variation Distance Uniform Distribution

Hello I am trying to solve the following but the answer is wrong and I cant seem to see my mistake. Question : Find the total variation distance between P = Unif([0,s]) and Q = Unif([0,t]) where 0 ...
0
votes
0answers
33 views

Relationship between KL divergence, JS divergence, and MMD?

What kind of relationship is there between the KL (Kullback-Leibler) divergence, JS (Jensen-Shannon) divergence, and MMD (maximum mean discrepancy)? I know that they all share a global minimum at $P=...
2
votes
1answer
196 views

KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
0
votes
1answer
210 views

How to achieve variational autoencoder (VAE) with unrestricted input?

For a normal VAE an input and a reconstruction with values in the range of $[0, 1]$ are expected. This is necessary since the log loss only makes sense for this range. If the input is not within $[0, ...
0
votes
2answers
71 views

KL-Divergence and the chain rule

I was trying to understand the mathematical proof of KL-Divergence when using the chain rule: $D(p(y|x)||q(y|x)) = D(p(x)||q(x)) + D(p(y|x)||q(y|x))$ And I'm a bit lost in the last step (https://www....
0
votes
1answer
21 views

Expectation of multivariate gaussian w.r.t. other multivariate gaussian

I want to calculate the Kullbach Leibler Divergence of two multivariate Gaussians as in KL divergence between two multivariate Gaussians. At one point one has to solve the following expression (...
1
vote
0answers
74 views

KL divergence between sample and true (multivariate normal) distribution

I was wondering, whether there is a possible interpretation of the KL-Divergence between sample and true distribution in terms of probabilities. E.g. given $P=\mathcal{N}\left(\mu,\Sigma\right)$ and $...
9
votes
1answer
118 views

Why does the Bayesian posterior concentrate around the minimiser of KL divergence?

Consider the Bayesian posterior $\theta\mid X$. Asymptotically, its maximum occurs at the MLE estimate $\hat \theta$, which just maximizes the likelihood $\operatorname{argmin}_\theta\, f_\theta(X)$. ...
1
vote
0answers
19 views

What scenario corresponds to choosing the “true distribution” $p$ in $\textsf{KL}(p\parallel q)$?

I understand that when you think about changing $q$ in the Kullback-Leibler divergence $\textsf{KL}(p\parallel q)$, this corresponds to trying to find the distribution that minimizes information loss ...
0
votes
0answers
105 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
0
votes
0answers
40 views
1
vote
0answers
63 views

compute the KL divergence between two datasets

I have two datasets $D1$ and $D2$ in two different feature spaces $\mathcal{X}_{1} \in \Re^{m}$ and $\mathcal{X}_{2} \in \Re^{n}$. Further assume that the datasets have different number of data points....
4
votes
1answer
50 views

KL divergence of models with both continuous and discrete variables

When $x$ is discrete, KL divergence is $D_{KL}(P||Q)=\sum\limits_{x}P(x)\log \frac{P(x)}{Q(x)}$, when $x$ is continuous, $D_{KL}(P||Q)=\int\limits_{x}p(x)\log \frac{p(x)}{q(x)}dx$. However, when the ...
1
vote
0answers
51 views

Relation between the AIC and the Kullback-Leibler Divergence

I am searching a formal derivation of the Akaike Information Criterion from the Kullback-Leibler Divergence. Can you show me one, or point me toward a book/article in which this is done? Here I set ...
0
votes
1answer
72 views

A different proof for KL divergence non-negativity

KL divergence's non-negativity can be proved in many ways. One could use the inequality $\log x \leq x - 1$ as a main step in the proof, another one could leverage the property of concave of the ...
1
vote
1answer
89 views

clustering with KL divergence

Recently I read some papers which related to clustering. The paper is https://xifengguo.github.io/papers/ICONIP17-DCEC.pdf In this paper, they calculate Loss function as KL Divergence, KL(P||Q). ...
0
votes
0answers
24 views

Are there two motivations for Bayesian information criteria?

Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score." Let the in-sample data be $y$ and the parameter ...
1
vote
1answer
50 views

Bayesian consistency in compact uncountable parameter space

Let $p(y_i \mid \theta)$ be the likelihood we are using of a single data point, $p(\theta)$ be the prior, and $f(y_i)$ the true distribution of the data. Also, let $\theta_0$ be the parameter that ...
0
votes
1answer
56 views

Expectation of KL between Categorical and samples from a Dirichlet

Is there a nice closed form expression for $\mathbb{E}_{\theta' \sim Dir(\alpha)} KL (Cat(x; \theta)|| Cat(x;\theta')$, where $Dir(\alpha)$ is the Dirichlet distribution with concentration parameters $...
2
votes
1answer
209 views

Kullback–Leibler Divergence of two 2-dimensional probability distributions

I need to calculate a KL divergence (and others) in the following question. But I have a hard time to understand the meaning of this syntax. Isn't P a function of (x,y)? I wasn't able to find any ...
1
vote
0answers
25 views

Conditional KL Divergence in Clustering Paper

I am trying to implement the following paper on Self-taught Clustering https://www.cse.ust.hk/~qyang/Docs/2008/dwyakicml.pdf. I have the following three co-clustering functions: where p̃(Z|x̃) is Z ...
2
votes
1answer
40 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
0
votes
0answers
12 views

How to measure the “variability” across a set of many (>>2) probability distributions?

Given a set of many of discrete probability distributions, is there a way I can efficiently calculate a metric that quantifies how different the entire set of these probability distributions are to ...
2
votes
1answer
44 views

KL-Diverence of Q(z|X) and P(z) in Variational Autoencoder (VAE)

I aim to understand how $D_{KL}[Q(z | X) || P(z)]$ can be converted to $\frac{1}{2} \sum_{k} (\Sigma(X) + \mu^{2}(X) - 1 - \log \Sigma(X))$, where $k$ is the dimension of the Gaussian distribution. ...
3
votes
0answers
148 views

Kullback-Leibler divergence with huge values? [closed]

I was playing with the KL divergence. My simple example is to calculate the divergence between two 2-dimensional normal distribution using PyTorch. The code for doing it is just below. I get an ...
0
votes
0answers
22 views

Confusion about maximising vs minimising (relative) entropy terminology and methods

I'm studying rules of inference for updating from a prior probability distribution to a posterior. One method for doing this is by maximising entropy, subject to constraints. I'm reading papers like ...
1
vote
0answers
8 views
3
votes
1answer
86 views

Can we apply KL divergence to the probability distributions on different domains?

When I was reading the original paper of t-SNE, I had an question whether or not we can apply KL divergence to the discrete probability distributions on different domains. In the paper, they measure ...
2
votes
1answer
118 views

Difference of notation between cross entropy and joint entropy

Although it is clear to me, how the two concepts differs, it has been difficult for me to find a notation that would make it clear, to which type of entropy we refer. From wikipedia, we can see that ...
0
votes
1answer
66 views

Kullback-Leibler Divergence as the class separation measure

I am trying to use KL divergence as the separation measure between the classes. I have the positive, negative samples for 2 distributions and want to adjust the algorithm parameters to get the best ...
1
vote
0answers
19 views

Solution of $D_{KL}(q_\phi(z)||p_\theta(z))$ of Gaussian case

Following is from the original paper of concept of VAE(variational autoencoder) by Kingma,Welling 2014 B. Solution of $D_{KL}(q_\phi(z)||p_\theta(z))$ of Gaussian case The variational lower bound (...
0
votes
1answer
168 views

KL Divergence with different domains

I want to calculate KL Divergence between a normal and an exponential r.v. i.e. $$D(P||Q) = ?\\ \;\; P=N(\mu,\sigma), \;\; Q=exp(\lambda)$$ My problem is that in this case the domains of the ...
3
votes
2answers
322 views
0
votes
0answers
28 views

Derivation of ELBO upon the Existence of Conditional Latent Variable Model

I am reading the recently published paper from DeepMind, "Neural Scene Representation and Rendering" and especially its "Supplementary Materials". Following is the page 1 and it's pretty hard for ...
1
vote
0answers
156 views

Finding the value of KL divergence to determine whether one distribution is distrinct from another?

Given the KL divergence value between 2 distributions, how should someone use this to determine whether the value is significant for the distributions $P$ and $Q$ to be different? One method I can ...
1
vote
0answers
49 views

Distance between angle distributions

I want to quantify the complexity of the street network of different cities. For each city I have the angle distribution of its streets. My hypothesis that the more complex the street network, the ...
2
votes
1answer
144 views

Why minimize the KL divergence from data to the model instead of KL divergence of model to data?

When minimizing the KL divergence in machine learning, why the expression of KL is from data to model, instead of from model to data ?
3
votes
1answer
641 views

KL divergence between which distributions could be infinity

I know that KL divergence measures difference between two probability distributions. My doubt is for which of the distributions it could become Infinity, putting it in another way, ...
5
votes
1answer
2k views

What is the difference Cross-entropy and KL divergence?

Both of Cross-entropy and KL divergence are tools to measure the distance between two probability distribution. What is the difference? $$ H(P,Q) = -\sum_x P(x)\log Q(x) $$ $$ KL(P | Q) = \sum_{x} P(...
0
votes
1answer
251 views

Rate of converge of KL-divergence to posterior

Suppose you have samples from some distribution P. You have a prior distribution Q, which represents your estimate of P, and assume for now that it's parameterized the same way as P. Upon observing ...
0
votes
1answer
108 views

Difference between Empirical distribution and Bernoulli distribution

I've been studying binary cross entropy error for binary classification weight optimization. From my knowledge, Cross entropy itself quantifies divergence between two probability distributions with ...
3
votes
2answers
262 views

Why Kullback-Leibler in Stochastic Neighbor Embedding

Stochastic Neighbor Embedding (and t-SNE) relies on Kullback-Leibler divergence between the point distributions in the original and the low-dimensional space. Why? Why not any other dissimilarity ...
1
vote
0answers
39 views

Choosing an appropriate statistical distance which punishes entropy

Problem Description: I have with me experimental statistics of a system and I wish to fit a theoretical model so that the computed statistics on the model fit the experimental ones. I am using an RBM ...
12
votes
3answers
3k views

What's the maximum value of Kullback-Leibler (KL) divergence

I am going to use KL divergence in my python code and I got this tutorial. On that tutorial, to implement KL divergence is quite simple. ...
4
votes
1answer
65 views

Comparing distribution A to B and C

I have three discrete probability distributions, A, B and C. They are all measuring P(X) under different circumstances. I suspect that A is more similar to B than it is to C. I know that I can compare ...