Linked Questions

0
votes
2answers
118 views

Mysteriously defined KL-divergence term [duplicate]

I am trying to re-create a variational autoencoder. The loss function has two terms: reconstruction loss and KL-divergence term. KL-divergence is defined as $$ D_{KL}(P||Q) = -\sum_{x\in X}{P(X)\log\...
66
votes
1answer
63k views

KL divergence between two multivariate Gaussians

I'm having trouble deriving the KL divergence formula assuming two multivariate normal distributions. I've done the univariate case fairly easily. However, it's been quite a while since I took math ...
19
votes
1answer
10k views

Deriving the KL divergence loss for VAEs

In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, ...
11
votes
2answers
6k views

KL Loss with a unit Gaussian

I've been implementing a VAE and I've noticed two different implementations online of the simplified univariate gaussian KL divergence. The original divergence as per here is $$ KL_{loss}=\log(\frac{\...
15
votes
3answers
4k views

How can I get feature importance for Gaussian Naive Bayes classifier?

I have a dataset consisting of 4 classes and around 200 features. I have implemented a Gaussian Naive Bayes classifier. I want now calculate the importance of each feature for each pair of classes ...
4
votes
2answers
7k views

How to use Kullback-leibler divergence if mean and standard deviation of of two Gaussian Distribution is provided?

With Apache Spark MLLib library I am trying to find Clusters within a dataset by using Gaussian Mixture Model (number cluster =3) . Now it returns 3 different values of mean and standard deviation. I ...
5
votes
1answer
4k views

Kullback-Leibler divergence of two normal distributions

I was recently trying to find a way to compute the KL-divergence between 2 populations that are normally distributed using the mean and variance of each population. But I found several different ...
3
votes
1answer
2k views

KL divergence between two bivariate Gaussian distribution

KL divergence between two multivariate Gaussians and univariate Gaussians have been discussed. I was wondering if there exists a simpler computation for the KL divergence between two bivariate ...
4
votes
1answer
705 views

KL divergence between an uninformative (?) Gaussian and a Gaussian

I have to calculate the KL divergence between a distribution $q$ and a prior distribution $p$, both of which are univariate Gaussians, i.e. $KL(q|p), q \sim \mathcal{N}(\mu, \sigma^2), p \sim \mathcal{...
2
votes
1answer
1k views

Understanding KL divergence between two univariate Gaussian distributions

I'm trying to understand KL divergence from this post on SE. I am following @ocram's answer, I understand the following : $\int \left[\log( p(x)) - log( q(x)) \right] p(x) dx$ $=\int \left[ -\frac{1}...
1
vote
1answer
448 views

Relationship between KL divergence and correlation

I know KL divergence tries to measure how different 2 probability distributions are. I know high correlation values between 2 sets of variables imply they are highly dependent on each other. Will ...
0
votes
1answer
177 views

KL divergence from PDF vs. mean and variance

I am trying to implement the KL divergence between two Gaussian distributions in Python. Since I have the mean and variance from both distributions, I was working with the following formula: $$ KL(p, ...
0
votes
0answers
195 views

Lognormal VAE Formulation

I'm looking at the following implementation of a VAE: https://github.com/jmtomczak/vae_vpflows/blob/master/models/VAE.py KL divergence is implemented as: ...
2
votes
1answer
153 views

what is -0.5 in VAE loss function with KL term

The VAE loss is composed of two terms: Reconstruction loss KLD loss in the implementation there is -0.5 applied to KLD loss. Kindly let me know what is this -0.5
0
votes
0answers
16 views

computing the mutual information of a mixture of Gaussians

How can one compute the mutual information of Gaussian mixture distribution \begin{equation} \mathrm{I}(X;C|Y)=H[C|Y]-H(C|X,Y)=-\int\int\sum_{C}P(X,C,Y)\log\frac{P(X,C|Y)}{P(C|Y)P(X|Y)}\mathrm{d}X\...