Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

0
votes
0answers
11 views

Kullback-Leibler divergence Loss With different-length vectors

I am new to KL Divergence Loss (and indeed all similar comparisons between discrete series data). The output of my network produces a series of tuples of a length that varies during training. The ...
1
vote
0answers
5 views

Markov type bound based on KL divergence?

Given two discrete distribution $p, q$ on some universe $U$, if I know they have a bounded KL divergence, say some number $c$, can I say anything about how much each point in the universe differs in ...
5
votes
1answer
135 views

Why Kullback-Leilbler divergence is a better metric for measuring distance between two probability distributions than squared error?

I know that KL-divergence is a metric that is more suitable when we want to measure the distance between numbers which a probability form. However, I am still confused what is the benefit of using KL-...
0
votes
0answers
15 views

A divergence that can be extended to logistic functions?

If I have data $\{(x_i, y_i)\}_{i=1}^n$ where the dependent variable is binary $(y_i = 0,1)$ I can model it using a logistic function: $$f(x; \alpha, \beta) = \frac{1}{1 + e^{-(\alpha + \beta x)}}$$ ...
2
votes
0answers
49 views

Relation Between Wasserstein Distance and Relative Entropy

Consider the Wasserstein metric of order one $W_1$ (aka the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and relative entropy and what this would mean intuitively. ...
0
votes
1answer
23 views

Can multivariate Gaussians KL divergence be a negative value?

I'm trying to find if two hidden neurons in RBF Network overlap with each other or not? It's an online classification problem, it means data come to our network one-by-one and then discard completely. ...
0
votes
1answer
51 views

KL divergence between gaussian and uniform distribution

Is the KL divergence not defined because uniform has bounded support and gaussian has unbounded support? How else can I calculate the distance of my gaussian to a 'maximum entropy' distribution if I ...
0
votes
0answers
15 views

Quantifying information loss (KL divergence?) between a multivariate and a univariate discrete distribution

Let's say I have n discrete variables, n1, n2, ... n_n, each with a different scale, and another discrete variable ...
1
vote
0answers
23 views

Running two MCMC chains in parallel while minimizing Kullback-Leibler divergence between both sample distributions

I want to sample from a distribution $p(X)$ with $X \in R^n$. However, I can only evaluate the likelihoods of $Z = AX$ and $Z = BX$ with $A,B \in R^{m \times n}$ and $m = n-1$. Now my idea is to run ...
2
votes
1answer
40 views

What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be minimised, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the maximisation of the corresponding ...
2
votes
0answers
42 views

Name/definition of $\int \log F(x) \cdot g(x)dx$?

We know that: $$-\int \log f(x) \cdot g(x)dx,$$ where $f$ and $g$ are density functions, is known as the cross entropy. Does $$-\int \log F(x) \cdot g(x)dx,$$ where $F$ is the cumulative ...
0
votes
2answers
24 views

Mysteriously defined KL-divergence term [duplicate]

I am trying to re-create a variational autoencoder. The loss function has two terms: reconstruction loss and KL-divergence term. KL-divergence is defined as $$ D_{KL}(P||Q) = -\sum_{x\in X}{P(X)\log\...
1
vote
0answers
67 views

Understanding KL divergence between two univariate Gaussian distributions

I'm trying to understand KL divergence from this post on SE. I am following @ocram's answer, I understand the following : $\int \left[\log( p(x)) - log( q(x)) \right] p(x) dx$ $=\int \left[ -\frac{1}...
1
vote
0answers
29 views

Approximating the Kullback-Leibler Divergence with a Laplace approximation

Suppose I wish to compute the (asymptotic) Kullback-Leibler Divergence (KLD) between the exact Bayesian posterior $$q_{n}(\theta|x_{1:n}) \propto \pi(\theta)\prod_{i=1}^n p(x_i|\theta)$$ and the ...
0
votes
0answers
16 views

Label smoothing formula

I recently came across this paper in section 3.2 it talks about label smoothing loss and how it's equivalent to s equivalent to adding the KL divergence between the uniform distribution u and the ...
3
votes
1answer
145 views

Does the Jensen-Shannon divergence maximise likelihood?

Minimising the KL divergence between your model distribution and the true data distribution is equivalent to maximising the (log-) likelihood. In machine learning, we often want to create a model ...
0
votes
0answers
30 views

Are there alternatives like KL divergence between continuous and discrete dist?

With KL divergence, I found it's impossible to see Is it possible to apply KL divergence between discrete and continuous distribution? . But I wanted to indicate using KL divergence, for example, ...
1
vote
0answers
28 views

skew G-Jensen-Shannon divergence between multivariate gaussian calculation discrepancy

I'm trying to calculate the Jensen-Shannon divergence between two multivariate Gaussians. I found a closed-form expression both for the KL divergence and JS divergence between two Gaussians in this ...
0
votes
0answers
15 views

Difference of mutual informations / Kullback-Leibler divergences for dependent arbitrary- and Gaussian random variables with similar second moments

Let $(Y_1, Y_2)$ be arbitrarily jointly distributed random variables, and let $(Y_{1,G}, Y_{1,G})$ be jointly distributed Gaussian random variables with the same mean and second moments as those of $(...
2
votes
1answer
41 views

On a mistake computing the Kullback Liebler Information Criterion

THE FRAMEWORK: Let $X_1$ be an observation from a normal random variable with mean zero and variance $\sigma^2$ and lets call the PDF $f(x)$. I want to minimize the Kullback Liebler Information ...
0
votes
0answers
43 views

Bounding the KL divergence of a non-invertible transform of distributions

Let $p$ and $q$ be the distributions of random variables $x_1$ and $x_2$, and consider $p'$ and $q'$ to be the distributions of $g(x_1)$ and $g(x_2)$. For an invertible function $g$, it's true that ...
0
votes
0answers
9 views

What should the form of error be on CrossEntropy or KL-divergence loss function across samples of distributions?

Suppose your model produces (discrete) probability distributions and you have some truth distributions you want to compare to. For each sample $i$, you can compute the loss as the KL divergence or ...
0
votes
0answers
29 views

When do these two definitions of KL-divergence match?

Suppose $P$ and $Q$ are two distributions on a space ${\cal H}$ (could be a subset of an infinite dimensional function space) with p.d.fs denoted by the same letter then one can define the $KL$ ...
0
votes
1answer
57 views
0
votes
1answer
72 views

Total Variation Distance Uniform Distribution

Hello I am trying to solve the following but the answer is wrong and I cant seem to see my mistake. Question : Find the total variation distance between P = Unif([0,s]) and Q = Unif([0,t]) where 0 ...
0
votes
0answers
74 views

Relationship between KL divergence, JS divergence, and MMD?

What kind of relationship is there between the KL (Kullback-Leibler) divergence, JS (Jensen-Shannon) divergence, and MMD (maximum mean discrepancy)? I know that they all share a global minimum at $P=...
2
votes
1answer
777 views

KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
0
votes
1answer
605 views

How to achieve variational autoencoder (VAE) with unrestricted input?

For a normal VAE an input and a reconstruction with values in the range of $[0, 1]$ are expected. This is necessary since the log loss only makes sense for this range. If the input is not within $[0, ...
0
votes
2answers
198 views

KL-Divergence and the chain rule

I was trying to understand the mathematical proof of KL-Divergence when using the chain rule: $D(p(y|x)||q(y|x)) = D(p(x)||q(x)) + D(p(y|x)||q(y|x))$ And I'm a bit lost in the last step (https://www....
0
votes
1answer
38 views

Expectation of multivariate gaussian w.r.t. other multivariate gaussian

I want to calculate the Kullbach Leibler Divergence of two multivariate Gaussians as in KL divergence between two multivariate Gaussians. At one point one has to solve the following expression (...
1
vote
0answers
142 views

KL divergence between sample and true (multivariate normal) distribution

I was wondering, whether there is a possible interpretation of the KL-Divergence between sample and true distribution in terms of probabilities. E.g. given $P=\mathcal{N}\left(\mu,\Sigma\right)$ and $...
9
votes
1answer
144 views

Why does the Bayesian posterior concentrate around the minimiser of KL divergence?

Consider the Bayesian posterior $\theta\mid X$. Asymptotically, its maximum occurs at the MLE estimate $\hat \theta$, which just maximizes the likelihood $\operatorname{argmin}_\theta\, f_\theta(X)$. ...
1
vote
0answers
20 views

What scenario corresponds to choosing the “true distribution” $p$ in $\textsf{KL}(p\parallel q)$?

I understand that when you think about changing $q$ in the Kullback-Leibler divergence $\textsf{KL}(p\parallel q)$, this corresponds to trying to find the distribution that minimizes information loss ...
0
votes
0answers
111 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
0
votes
0answers
95 views
1
vote
0answers
146 views

compute the KL divergence between two datasets

I have two datasets $D1$ and $D2$ in two different feature spaces $\mathcal{X}_{1} \in \Re^{m}$ and $\mathcal{X}_{2} \in \Re^{n}$. Further assume that the datasets have different number of data points....
4
votes
1answer
75 views

KL divergence of models with both continuous and discrete variables

When $x$ is discrete, KL divergence is $D_{KL}(P||Q)=\sum\limits_{x}P(x)\log \frac{P(x)}{Q(x)}$, when $x$ is continuous, $D_{KL}(P||Q)=\int\limits_{x}p(x)\log \frac{p(x)}{q(x)}dx$. However, when the ...
1
vote
0answers
80 views

Relation between the AIC and the Kullback-Leibler Divergence

I am searching a formal derivation of the Akaike Information Criterion from the Kullback-Leibler Divergence. Can you show me one, or point me toward a book/article in which this is done? Here I set ...
0
votes
1answer
94 views

A different proof for KL divergence non-negativity

KL divergence's non-negativity can be proved in many ways. One could use the inequality $\log x \leq x - 1$ as a main step in the proof, another one could leverage the property of concave of the ...
1
vote
1answer
185 views

clustering with KL divergence

Recently I read some papers which related to clustering. The paper is https://xifengguo.github.io/papers/ICONIP17-DCEC.pdf In this paper, they calculate Loss function as KL Divergence, KL(P||Q). ...
0
votes
0answers
25 views

Are there two motivations for Bayesian information criteria?

Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score." Let the in-sample data be $y$ and the parameter ...
1
vote
1answer
51 views

Bayesian consistency in compact uncountable parameter space

Let $p(y_i \mid \theta)$ be the likelihood we are using of a single data point, $p(\theta)$ be the prior, and $f(y_i)$ the true distribution of the data. Also, let $\theta_0$ be the parameter that ...
0
votes
1answer
77 views

Expectation of KL between Categorical and samples from a Dirichlet

Is there a nice closed form expression for $\mathbb{E}_{\theta' \sim Dir(\alpha)} KL (Cat(x; \theta)|| Cat(x;\theta')$, where $Dir(\alpha)$ is the Dirichlet distribution with concentration parameters $...
2
votes
1answer
274 views

Kullback–Leibler Divergence of two 2-dimensional probability distributions

I need to calculate a KL divergence (and others) in the following question. But I have a hard time to understand the meaning of this syntax. Isn't P a function of (x,y)? I wasn't able to find any ...
1
vote
0answers
28 views

Conditional KL Divergence in Clustering Paper

I am trying to implement the following paper on Self-taught Clustering https://www.cse.ust.hk/~qyang/Docs/2008/dwyakicml.pdf. I have the following three co-clustering functions: where p̃(Z|x̃) is Z ...
2
votes
1answer
49 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
0
votes
0answers
13 views

How to measure the “variability” across a set of many (>>2) probability distributions?

Given a set of many of discrete probability distributions, is there a way I can efficiently calculate a metric that quantifies how different the entire set of these probability distributions are to ...
2
votes
1answer
52 views

KL-Diverence of Q(z|X) and P(z) in Variational Autoencoder (VAE)

I aim to understand how $D_{KL}[Q(z | X) || P(z)]$ can be converted to $\frac{1}{2} \sum_{k} (\Sigma(X) + \mu^{2}(X) - 1 - \log \Sigma(X))$, where $k$ is the dimension of the Gaussian distribution. ...
3
votes
0answers
219 views

Kullback-Leibler divergence with huge values? [closed]

I was playing with the KL divergence. My simple example is to calculate the divergence between two 2-dimensional normal distribution using PyTorch. The code for doing it is just below. I get an ...
0
votes
0answers
22 views

Confusion about maximising vs minimising (relative) entropy terminology and methods

I'm studying rules of inference for updating from a prior probability distribution to a posterior. One method for doing this is by maximising entropy, subject to constraints. I'm reading papers like ...