# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

296 questions
Filter by
Sorted by
Tagged with
37 views

18 views

### “Compressing” or downsampling a discrete probability distribution

I have a discrete probability distribution $P$ which I obtained by applying a softmax transformation, with an automatically-derived exponent $-\beta$, to a set of measurements (potentially large, in ...
100 views

### Expectation of the log-likelihood under the posterior

Suppose $L(X \mid \theta)$ is a likelihood function, i.e., a probability distribution over $X \in \mathcal{X}$ indexed by a parameter $\theta \in \Theta$. Suppose further we have a prior $\pi(\theta)$,...
33 views

### Measure the distance between two probability transition matrices

I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
16 views

25 views

### How to prove KL(q(z)|p(z)) = E_q(z) [ log f(z) ] where f is the CDF of p?

As title. It was used in https://arxiv.org/abs/1905.10549 without proving.
15 views

### Kullback-Leibler divergence Loss With different-length vectors

I am new to KL Divergence Loss (and indeed all similar comparisons between discrete series data). The output of my network produces a series of tuples of a length that varies during training. The ...
8 views

### Markov type bound based on KL divergence?

Given two discrete distribution $p, q$ on some universe $U$, if I know they have a bounded KL divergence, say some number $c$, can I say anything about how much each point in the universe differs in ...
398 views

### Why is Kullback-Leilbler divergence a better metric for measuring distance between two probability distributions than squared error? [duplicate]

I know that KL-divergence is a metric that is more suitable when we want to measure the distance between numbers which a probability form. However, I am still confused what is the benefit of using KL-...
18 views

### A divergence that can be extended to logistic functions?

If I have data $\{(x_i, y_i)\}_{i=1}^n$ where the dependent variable is binary $(y_i = 0,1)$ I can model it using a logistic function: $$f(x; \alpha, \beta) = \frac{1}{1 + e^{-(\alpha + \beta x)}}$$ ...
65 views

### Relation Between Wasserstein Distance and Relative Entropy

Consider the Wasserstein metric of order one $W_1$ (aka the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and relative entropy and what this would mean intuitively. ...
41 views

### Can multivariate Gaussians KL divergence be a negative value?

I'm trying to find if two hidden neurons in RBF Network overlap with each other or not? It's an online classification problem, it means data come to our network one-by-one and then discard completely. ...
137 views

### KL divergence between gaussian and uniform distribution

Is the KL divergence not defined because uniform has bounded support and gaussian has unbounded support? How else can I calculate the distance of my gaussian to a 'maximum entropy' distribution if I ...
18 views

### Quantifying information loss (KL divergence?) between a multivariate and a univariate discrete distribution

Let's say I have n discrete variables, n1, n2, ... n_n, each with a different scale, and another discrete variable ...
26 views

### Running two MCMC chains in parallel while minimizing Kullback-Leibler divergence between both sample distributions

I want to sample from a distribution $p(X)$ with $X \in R^n$. However, I can only evaluate the likelihoods of $Z = AX$ and $Z = BX$ with $A,B \in R^{m \times n}$ and $m = n-1$. Now my idea is to run ...
79 views

### What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be minimised, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the maximisation of the corresponding ...
42 views

### Name/definition of $\int \log F(x) \cdot g(x)dx$?

We know that: $$-\int \log f(x) \cdot g(x)dx,$$ where $f$ and $g$ are density functions, is known as the cross entropy. Does $$-\int \log F(x) \cdot g(x)dx,$$ where $F$ is the cumulative ...
29 views

120 views

### Label smoothing formula

I recently came across this paper in section 3.2 it talks about label smoothing loss and how it's equivalent to s equivalent to adding the KL divergence between the uniform distribution u and the ...
190 views

### Does the Jensen-Shannon divergence maximise likelihood?

Minimising the KL divergence between your model distribution and the true data distribution is equivalent to maximising the (log-) likelihood. In machine learning, we often want to create a model ...
32 views

### Are there alternatives like KL divergence between continuous and discrete dist?

With KL divergence, I found it's impossible to see Is it possible to apply KL divergence between discrete and continuous distribution? . But I wanted to indicate using KL divergence, for example, ...
45 views

### skew G-Jensen-Shannon divergence between multivariate gaussian calculation discrepancy

I'm trying to calculate the Jensen-Shannon divergence between two multivariate Gaussians. I found a closed-form expression both for the KL divergence and JS divergence between two Gaussians in this ...
15 views

2k views

### KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
881 views

177 views

### Why does the Bayesian posterior concentrate around the minimiser of KL divergence?

Consider the Bayesian posterior $\theta\mid X$. Asymptotically, its maximum occurs at the MLE estimate $\hat \theta$, which just maximizes the likelihood $\operatorname{argmin}_\theta\, f_\theta(X)$. ...
20 views

### What scenario corresponds to choosing the “true distribution” $p$ in $\textsf{KL}(p\parallel q)$?

I understand that when you think about changing $q$ in the Kullback-Leibler divergence $\textsf{KL}(p\parallel q)$, this corresponds to trying to find the distribution that minimizes information loss ...
113 views

### rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
204 views

### cont. KL divergence in python

...
I have two datasets $D1$ and $D2$ in two different feature spaces $\mathcal{X}_{1} \in \Re^{m}$ and $\mathcal{X}_{2} \in \Re^{n}$. Further assume that the datasets have different number of data points....
When $x$ is discrete, KL divergence is $D_{KL}(P||Q)=\sum\limits_{x}P(x)\log \frac{P(x)}{Q(x)}$, when $x$ is continuous, $D_{KL}(P||Q)=\int\limits_{x}p(x)\log \frac{p(x)}{q(x)}dx$. However, when the ...