Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

Minimization of Midpoint Kullback-Leibler Divergence

Consider the following problems: $$ \arg\min \limits_{\gamma \in \mathcal{\Gamma}} \; \mbox{KL}\Big(\gamma \; \Big\| \;\frac{\gamma + \nu}{2} \Big) ,$$ $$ \arg\min_{\gamma \in \mathcal{\Gamma}} \; \...
0
votes
0answers
9 views

Kullback Leibler divergence for convolutional variational autoencoders

I have a convnet architecture which downsamples to 256x256 input to 16x16 = 256 latent variables. Training with a batch size of 10, the KL term calculates the sum over the the latents first and then ...
0
votes
0answers
25 views

Why is forward Kullback Leibler Divergence mean seeking?

I have a course on Information theory, in the which we talk about forward KLD in order to approximate pdfs. There is an example that's the same example as on this blog : https://towardsdatascience.com/...
1
vote
0answers
29 views

KL divergence from a family of distributions

The KL divergence between two probability measures $q,p$ is $$KL(q(x)||p(x)) = - \int_{\mathcal{X}} q(x) \log\frac{p(x)}{q(x)} dx.$$ What's the KL divergence between a measure and a set of measures? ...
0
votes
0answers
16 views

Laplace distribution as an Exponential Distribution and Minimizitaion of KL Divergence

In the context of Expectation Propagation [Minka's thesis-2001], I would like to approximate an unknown distribution with a Laplace distribution. This can be solved by minimizing KL-Divergence. In ...
1
vote
0answers
66 views

Moment Matching for a Laplace Distribution

I have this derivative It belongs to this paper. In the paper, they are trying to model a lightweight bayesian deep neural network by having the distributions on only the activation functions. They ...
0
votes
1answer
14 views

Entropy of gaussian mixture

Does the entropy of a gaussian mixture depend on its means? It is not the case for a single Gaussian and when the components of the mixture are far spread out, we can approximate the entropy by a ...
0
votes
0answers
31 views

What is extropy?

While searching for the answer to a statistics problem, I came across the term extropy, which was - and to a large extent still is - unfamiliar to me. A quick google search revealed that it is the &...
1
vote
1answer
63 views

Relationship between cross entropy and average negative log likelihood

I'm trying to understand some machine learning theory background: specifically, the relationship between cross entropy loss and "negative log likelihood". To start, I already fully ...
0
votes
1answer
17 views

Multivariate KL Divergence - does data ordering matter?

I have two multivariate probability distributions, empirically observed. Here's some artificial data in two dimensions (with each value representing the number of events that occurred in that grid ...
0
votes
0answers
24 views

Calculate KL Divergence given discrete points in a differentiable way

I have an input dataset $x$ drawn from an underlying distribution $P(x)$. $x = [x_1, x_2, ..., x_N]$ is a collection of $N$ points. I pass this into a neural network and it outputs a new dataset $\hat{...
0
votes
0answers
22 views

Upper bound on KL divergence for multinomial distributions

Suppose $\mathbb{P}$ is Multinomial$(1; p_1, p_2, \ldots, p_L)$ and $\mathbb{Q}$ is Multinomial$(1; q_1, q_2, \ldots, q_L)$. Assume that the difference between the probability masses are bounded, i.e. ...
0
votes
0answers
16 views

Why is the displacement term in my implementation of multivariate Kullback Leibler Divergence negative?

I'm trying to optimize my custom implementation of a Faster RCNN that provides ellipse outputs per detected instance. To achieve this, I've implemented a multivariate KLD function for PyTorch that I'...
1
vote
0answers
44 views

Why is reverse KL more suited for data generation

Here goes a first question! In a paper I'm reading in the context of GAN's (WGAN in particular) I came across the following quote when the authors discuss KL divergence: while maximum likelihood ...
2
votes
0answers
27 views

Why does Kullback–Leibler divergence measure information loss when approximating a probability distribution?

I've encountered a sentence: In information theory, Kullback–Leibler divergence is regarded as a measure of the information lost when probability distribution Q is used to approximate a true ...
0
votes
0answers
4 views

Is label smoothing equivalent to adding a KL divergence term or a cross entropy term?

In the context of cross-entropy loss objectives for neural networks, I tend to think of label smoothing from the standpoint of directly manipulating the labels. This blog post explains how doing so is ...
0
votes
0answers
18 views

How do we get from entropy to KL divergence in this paper?

I'm reading through Regularizing Neural Networks By Penalizing Confident Output Distributions and I'm stuck on the equation in section 3.2. It's not clear to me at all that the self-entropy of the ...
0
votes
0answers
22 views

KL Divergence of Empirical Distribution

Consider $n$ samples from a distribution $P$, given by $X_1,\ldots, X_n \sim P$ and denote $\hat{P}_n$ as the empirical distribution from the samples. What is the value (or rate of convergence) for ...
1
vote
1answer
31 views

Understanding math notation in infoGAN paper

I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I ...
2
votes
1answer
143 views

Does the Bhattacharyya distance and KL divergence measure the same thing?

I'm currently studying about the two methods in the title. Does the Bhattacharyya distance and KL divergence measure the same thing, but in a different way? I know the things that Bhattacharyya ...
1
vote
0answers
54 views

Why KL divergence fails to approximate the means of distributions? [closed]

We have two distributions, $P$ and $Q$ such that $P$ is our input distribution and $Q$ is our target distribution. The formulation of $KL = \mathbb{E}_{P}\left[\log\frac{P}{Q}\right]$ allows us to ...
1
vote
0answers
24 views

Net not learning in MINST with peculiar cost function

I've been 12 hours non stop trying to solve this. Defeated, I came here to see if someone can help: I'm training a LeNet architecture with MNIST in Pytorch, using the loss function from eq(5) this ...
0
votes
0answers
22 views

Significance testing for Jensen–Shannon divergence?

The Jensen-Shannon divergence (JSD) measures the (dis)similarity between multiple probability distributions. How can one determine whether the JSD of (a pair of, or multiple) distributions is ...
4
votes
1answer
153 views

KL divergence for joint probability distributions?

I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the Kullback–Leibler ...
1
vote
0answers
19 views

How to obtain uncertainty estimate around MLE parameters based on KL-Divergence?

Suppose I know some true distribution $S(x)$, and I have a method of approximating $S$ based on a transformation of another distribution $G(x|\theta)$. We denote the approximation as $S^*(x|\theta)$. ...
0
votes
0answers
12 views

Scaling [0,1] values so that their l1-norm is not uniform?

Let's say I have am sampling of $k$ values uniformly sampled from [0, 1], call the sample $X$. I would like to apply a transformation to them $f$, such that when I apply the $\ell_1$ norm to $f(X)$, ...
1
vote
0answers
119 views

KL divergence between two multivariate gaussians where $p$ is $N(\mu, I)$

We know if we try to get $D_{KL}(q||p)$, where $p$ is a standard normal distribution, so mean is 0, variance is the identity matrix, and $q$ is a multivariate normal distribution, it can be calculated ...
4
votes
2answers
76 views

Does maximizing Jensen–Shannon divergence maximize Kullback–Leibler divergence?

Does maximizing the Jensen–Shannon divergence $D_{\mathrm{JS}}(P \parallel Q)$ maximize the Kullback–Leibler divergence $D_{\mathrm{KL}}(P \parallel Q)$? If so, I'd like to be able to show that it ...
3
votes
1answer
189 views

Why use KL-Divergence as loss over MLE?

I have came across this statement several time now Maximizing likelihood is equivalent to minimizing KL-Divergence (Sources: Kullback–Leibler divergence and Maximum likelihood as minimizing the ...
2
votes
0answers
18 views

What exactly is the point of computing a lower bound for the log partition function in variational methods in probabilistic graphical models?

Variational methods are applied when we are interested in a probability distribution $P$ but only have a tractably computable unnormalized form $\tilde{P}$ of $P$. Knowing the partition function $Z = \...
4
votes
2answers
91 views

Why is KL divergence between a standard normal, and normal distribution 0 if standard deviation is 0.37?

I wanted to see a graph about how KL divergence between a standard normal distribution, and any other normal distributions with 0 mean, and standard deviation being $x$ varies. I mostly need it for ...
0
votes
1answer
35 views

Discrete KL Divergence with decreasing bin width

I'm familiar with the definition of the KL divergence between two discrete distributions $D_{KL} = D_{KL}\big({\it P}(A) || {\it Q}(B)\big)=\sum_{j=1}^{n} {\it P}(A=a_{j}) \log \Big( \cfrac{{\it P}(A=...
1
vote
0answers
97 views

Chain rule for KL divergence, conditional measures

The chain rule for KL divergence is widely seen in the theoretical machine learning literature and generally referenced to [1, Theorem 2.5.3]: $$ \text{KL}[p(x, y) \mid q(x, y)]= \text{KL}[p(x) \mid q(...
0
votes
0answers
28 views

Binary cross entropy and relationship with cross entropy function or KL divergence

I am learning GAN right now and I don't understand why the cost of the original GAN function will cause mode clapse and many problems. I did some research and notice it has something to do with KL ...
2
votes
1answer
63 views

Kullback-Leibler divergence between multivariate t and the multivariate normal?

I want to calculate the Kullback Leibler divergence between a multivariate t distribution and a multivariate normal distribution, for different values of the degrees of freedom $\nu$. However, this ...
1
vote
0answers
34 views

KL divergence and Wasserstein distance

While reading the paper https://arxiv.org/pdf/1903.11780.pdf, I have some confusion parts as below: The KL divergence is not only problematic for representation learning due to the statistical ...
2
votes
1answer
33 views

JS-divergence vs KL-divergence as objective for GAN?

I would love to get some help with that interview question I failed to answer correctly: In what sense is the JS-divergence prefferable over KL-divergence $D_{KL}(p_{data}||p_{model})$ as an objective ...
1
vote
1answer
20 views

Proving that JSD is symmetric?

How can I prove that the Jensen–Shannon divergence (https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence) is symmetric? Thanks!
0
votes
1answer
55 views

How can I prove that KL-divergence is not symmetric?

How can I prove that KL-divergence is not symmetric? https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence Thanks a lot!
1
vote
1answer
31 views

How do measure how different two policies are?

I have two agents that both follow a baseline behavioral policy pi(a|s). If I then modify the state-action distribution for the two agents (resulting in two new policies), is there a standard measure ...
1
vote
0answers
37 views

Why is the Kullback-Leibler divergence defined with a negative sign?

I am aware of Gibbs' inequality, but I still want to know why the Kullback-Leibler divergence is defined with a negative sign. Here is my reasoning so far: Let $X$ be a Bernoulli random variable with $...
0
votes
1answer
63 views

KL divergence from PDF vs. mean and variance

I am trying to implement the KL divergence between two Gaussian distributions in Python. Since I have the mean and variance from both distributions, I was working with the following formula: $$ KL(p, ...
1
vote
1answer
71 views

KL Divergence Asymmetry at Zero

So, I'm aware that the KL divergence asymmetric, and furthermore that it has to be asymmetric in order to be compatible with Bayes' Law. But there's one particular asymmetry that confuses me. Let $$KL(...
3
votes
1answer
81 views

Is there a G-test equivalent for continuous variables?

The G-test is similar to the chi-square test for goodness of fit. It is proportional to the kl-divergence. I am wondering if there is a similar test that is applicable to continuous variables. Since ...
1
vote
0answers
34 views

Can a likelihood's relative entropy be related to its predictive accuracy?

Suppose I have some prior $\pi(\theta)$, from which I draw $N$ samples, each having parameter $\theta_i$. These $\theta_i$'s are known to me. Suppose that one of these samples (unknown to me which) ...
1
vote
0answers
135 views

Variational Autoencoder: balance KL-Divergence and ReconstructionLoss

I'm facing some problems working with VAE. They are pretty similar than the ones at this thread: Balancing Reconstruction vs KL Loss Variational Autoencoder but I still have the problems and I don't ...
0
votes
0answers
12 views

Can divergences for probability distributions also be used for regular distributions? [duplicate]

As an example the Kullback Leibler divergence is used to compare two probability distributions. However if you have distributions that aren't described by probabilities, can you still apply such a ...
1
vote
0answers
23 views

Why optimizing the difference between $q(z|x)$, and $p(z|x)$ makes the latent variables “complete”?

In the past month I've spent most of my time digging deep into how neural networks work, from the basic idea (to estimate the true posterior $p(z|x)$, we create a variational posterior $q(z|x)$ - the ...
1
vote
0answers
27 views

How do $E_q[p(x|z)]$ becomes the cross entropy between $x$, and $p(x|z)$?

In variational autoencoders we get the difference between $q_\theta(z|x)$, and intractable $p_\phi(z|x)$ by calculating the ELBO with respect to their parameters by breaking it down to: $$E_q[p_\phi(x|...
1
vote
1answer
17 views

Is applying KL divergence to two fuzzy cluster membership assignments appropriate?

Deep Embedded Clustering, originally by Xie et al. and in a question on CV as well, assigns cluster memberships with a $t$-distribution, hardens them with a second power, and computes the KL ...

1
2 3 4 5
9