All Questions

Filter by
Sorted by
Tagged with
0
votes
1answer
37 views

KL divergence of a uniform prior and a custom posterior

So I was reading the Google's paper on VQ-VAE and have stumbled upon the derivation of KL divergence of the uniform prior and the given distribution: $$q(z=k \mid x)=\left\{\begin{array}{ll}{1} & ...
2
votes
1answer
54 views

What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be minimised, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the maximisation of the corresponding ...
0
votes
1answer
72 views
2
votes
1answer
1k views

KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
0
votes
1answer
720 views

How to achieve variational autoencoder (VAE) with unrestricted input?

For a normal VAE an input and a reconstruction with values in the range of $[0, 1]$ are expected. This is necessary since the log loss only makes sense for this range. If the input is not within $[0, ...
0
votes
0answers
111 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
2
votes
1answer
52 views

KL-Diverence of Q(z|X) and P(z) in Variational Autoencoder (VAE)

I aim to understand how $D_{KL}[Q(z | X) || P(z)]$ can be converted to $\frac{1}{2} \sum_{k} (\Sigma(X) + \mu^{2}(X) - 1 - \log \Sigma(X))$, where $k$ is the dimension of the Gaussian distribution. ...
2
votes
1answer
2k views

Deriving the KL divergence loss for VAEs

In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, ...
8
votes
2answers
2k views

KL Loss with a unit Gaussian

I've been implementing a VAE and I've noticed two different implementations online of the simplified univariate gaussian KL divergence. The original divergence as per here is $$ KL_{loss}=\log(\frac{\...
1
vote
0answers
228 views

Derivation of autoencoders in backpropagation

I'm following the basics of autoencoders here: http://ufldl.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity Here are some of the important parts: But I don't understand the last part: Why is ...