# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

296 questions
Filter by
Sorted by
Tagged with
0answers
37 views

0answers
18 views

### “Compressing” or downsampling a discrete probability distribution

I have a discrete probability distribution $P$ which I obtained by applying a softmax transformation, with an automatically-derived exponent $-\beta$, to a set of measurements (potentially large, in ...
2answers
100 views

### Expectation of the log-likelihood under the posterior

Suppose $L(X \mid \theta)$ is a likelihood function, i.e., a probability distribution over $X \in \mathcal{X}$ indexed by a parameter $\theta \in \Theta$. Suppose further we have a prior $\pi(\theta)$,...
0answers
33 views

### Measure the distance between two probability transition matrices

I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
0answers
16 views

0answers
25 views

### How to prove KL(q(z)|p(z)) = E_q(z) [ log f(z) ] where f is the CDF of p?

As title. It was used in https://arxiv.org/abs/1905.10549 without proving.
0answers
15 views

### Kullback-Leibler divergence Loss With different-length vectors

I am new to KL Divergence Loss (and indeed all similar comparisons between discrete series data). The output of my network produces a series of tuples of a length that varies during training. The ...
0answers
8 views

### Markov type bound based on KL divergence?

Given two discrete distribution $p, q$ on some universe $U$, if I know they have a bounded KL divergence, say some number $c$, can I say anything about how much each point in the universe differs in ...
1answer
398 views

### Why is Kullback-Leilbler divergence a better metric for measuring distance between two probability distributions than squared error? [duplicate]

I know that KL-divergence is a metric that is more suitable when we want to measure the distance between numbers which a probability form. However, I am still confused what is the benefit of using KL-...
0answers
18 views

### A divergence that can be extended to logistic functions?

If I have data $\{(x_i, y_i)\}_{i=1}^n$ where the dependent variable is binary $(y_i = 0,1)$ I can model it using a logistic function: $$f(x; \alpha, \beta) = \frac{1}{1 + e^{-(\alpha + \beta x)}}$$ ...
0answers
65 views

### Relation Between Wasserstein Distance and Relative Entropy

Consider the Wasserstein metric of order one $W_1$ (aka the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and relative entropy and what this would mean intuitively. ...
1answer
41 views

### Can multivariate Gaussians KL divergence be a negative value?

I'm trying to find if two hidden neurons in RBF Network overlap with each other or not? It's an online classification problem, it means data come to our network one-by-one and then discard completely. ...
1answer
137 views

### KL divergence between gaussian and uniform distribution

Is the KL divergence not defined because uniform has bounded support and gaussian has unbounded support? How else can I calculate the distance of my gaussian to a 'maximum entropy' distribution if I ...
0answers
18 views

### Quantifying information loss (KL divergence?) between a multivariate and a univariate discrete distribution

Let's say I have n discrete variables, n1, n2, ... n_n, each with a different scale, and another discrete variable ...
0answers
26 views

### Running two MCMC chains in parallel while minimizing Kullback-Leibler divergence between both sample distributions

I want to sample from a distribution $p(X)$ with $X \in R^n$. However, I can only evaluate the likelihoods of $Z = AX$ and $Z = BX$ with $A,B \in R^{m \times n}$ and $m = n-1$. Now my idea is to run ...
1answer
79 views

### What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be minimised, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the maximisation of the corresponding ...
0answers
42 views

### Name/definition of $\int \log F(x) \cdot g(x)dx$?

We know that: $$-\int \log f(x) \cdot g(x)dx,$$ where $f$ and $g$ are density functions, is known as the cross entropy. Does $$-\int \log F(x) \cdot g(x)dx,$$ where $F$ is the cumulative ...
2answers
29 views

1answer
120 views

### Label smoothing formula

I recently came across this paper in section 3.2 it talks about label smoothing loss and how it's equivalent to s equivalent to adding the KL divergence between the uniform distribution u and the ...
1answer
190 views

### Does the Jensen-Shannon divergence maximise likelihood?

Minimising the KL divergence between your model distribution and the true data distribution is equivalent to maximising the (log-) likelihood. In machine learning, we often want to create a model ...
0answers
32 views

### Are there alternatives like KL divergence between continuous and discrete dist?

With KL divergence, I found it's impossible to see Is it possible to apply KL divergence between discrete and continuous distribution? . But I wanted to indicate using KL divergence, for example, ...
0answers
45 views

### skew G-Jensen-Shannon divergence between multivariate gaussian calculation discrepancy

I'm trying to calculate the Jensen-Shannon divergence between two multivariate Gaussians. I found a closed-form expression both for the KL divergence and JS divergence between two Gaussians in this ...
0answers
15 views

1answer
2k views

### KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
1answer
881 views

1answer
177 views

### Why does the Bayesian posterior concentrate around the minimiser of KL divergence?

Consider the Bayesian posterior $\theta\mid X$. Asymptotically, its maximum occurs at the MLE estimate $\hat \theta$, which just maximizes the likelihood $\operatorname{argmin}_\theta\, f_\theta(X)$. ...
0answers
20 views

### What scenario corresponds to choosing the “true distribution” $p$ in $\textsf{KL}(p\parallel q)$?

I understand that when you think about changing $q$ in the Kullback-Leibler divergence $\textsf{KL}(p\parallel q)$, this corresponds to trying to find the distribution that minimizes information loss ...
0answers
113 views

### rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
0answers
204 views

...
0answers
208 views

### compute the KL divergence between two datasets

I have two datasets $D1$ and $D2$ in two different feature spaces $\mathcal{X}_{1} \in \Re^{m}$ and $\mathcal{X}_{2} \in \Re^{n}$. Further assume that the datasets have different number of data points....
1answer
93 views

### KL divergence of models with both continuous and discrete variables

When $x$ is discrete, KL divergence is $D_{KL}(P||Q)=\sum\limits_{x}P(x)\log \frac{P(x)}{Q(x)}$, when $x$ is continuous, $D_{KL}(P||Q)=\int\limits_{x}p(x)\log \frac{p(x)}{q(x)}dx$. However, when the ...