Skip to main content
113 votes

How does the reparameterization trick for VAEs work and why is it important?

Assume we have a normal distribution $q$ that is parameterized by $\theta$, specifically $q_{\theta}(x) = N(\theta,1)$. We want to solve the below problem $$ \text{min}_{\theta} \quad E_q[x^2] $$ This ...
goker's user avatar
  • 1,549
111 votes

How does the reparameterization trick for VAEs work and why is it important?

After reading through Kingma's NIPS 2015 workshop slides, I realized that we need the reparameterization trick in order to backpropagate through a random node. Intuitively, in its original form, VAEs ...
David Dao's user avatar
  • 2,834
58 votes
Accepted

What are variational autoencoders and to what learning tasks are they used?

Even though variational autoencoders (VAEs) are easy to implement and train, explaining them is not simple at all, because they blend concepts from Deep Learning and Variational Bayes, and the Deep ...
DeltaIV's user avatar
  • 18.1k
52 votes
Accepted

Deriving the KL divergence loss for VAEs

The encoder distribution is $q(z|x)=\mathcal{N}(z|\mu(x),\Sigma(x))$ where $\Sigma=\text{diag}(\sigma_1^2,\ldots,\sigma^2_n)$. The latent prior is given by $p(z)=\mathcal{N}(0,I)$. Both are ...
user3658307's user avatar
  • 2,214
50 votes

Building an autoencoder in Tensorflow to surpass PCA

Here is the key figure from the 2006 Science paper by Hinton and Salakhutdinov: It shows dimensionality reduction of the MNIST dataset ($28\times 28$ black and white images of single digits) from the ...
amoeba's user avatar
  • 106k
48 votes

When should I use a variational autoencoder as opposed to an autoencoder?

VAE is a framework that was proposed as a scalable way to do variational EM (or variational inference in general) on large datasets. Although it has an AE like structure, it serves a much larger ...
TenaliRaman's user avatar
  • 3,801
47 votes
Accepted

What is the "capacity" of a machine learning model?

Capacity is an informal term. It's very close (if not a synonym) for model complexity. It's a way to talk about how complicated a pattern or relationship a model can express. You could expect a model ...
shimao's user avatar
  • 26.3k
35 votes

When should I use a variational autoencoder as opposed to an autoencoder?

The standard autoencoder can be illustrated using the following graph: As stated in the previous answers it can be viewed as just a nonlinear extension of PCA. But compared to the variational ...
Lerner Zhang's user avatar
  • 6,768
34 votes

How does the reparameterization trick for VAEs work and why is it important?

A reasonable example of the mathematics of the "reparameterization trick" is given in goker's answer, but some motivation could be helpful. (I don't have permissions to comment on that answer; thus ...
Seth Bruder's user avatar
34 votes
Accepted

How to weight KLD loss vs reconstruction loss in variational auto-encoder?

For anyone stumbling on this post also looking for an answer, this twitter thread has added a lot of very useful insight. Namely: beta-VAE: Learning Basic Visual Concepts with a Constrained ...
memo's user avatar
  • 1,019
31 votes
Accepted

Loss function for autoencoders

I think the best answer to this is that the cross-entropy loss function is just not well-suited to this particular task. In taking this approach, you are essentially saying the true MNIST data is ...
nlml's user avatar
  • 982
24 votes
Accepted

Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

I don't believe there's some kind of deep, meaningful rationale at play here - it's a showcase example running on MNIST, it's pretty error-tolerant. Optimizing for MSE means your generated output ...
jkm's user avatar
  • 2,165
24 votes
Accepted

How should I intuitively understand the KL divergence loss in variational autoencoders?

The KL divergence tells us how well the probability distribution Q approximates the probability distribution P by calculating the cross-entropy minus the entropy. Intuitively, you can think of that as ...
zoozoo's user avatar
  • 428
23 votes

How does the reparameterization trick for VAEs work and why is it important?

Let me explain first, why do we need Reparameterization trick in VAE. VAE has encoder and decoder. Decoder randomly samples from true posterior Z~ q(z∣ϕ,x). To implement encoder and decoder as a ...
Sherlock's user avatar
  • 359
21 votes

What're the differences between PCA and autoencoder?

The currently accepted answer by @bayerj states that the weights of a linear autoencoder span the same subspace as the principal components found by PCA, but they are not the same vectors. In ...
DeltaIV's user avatar
  • 18.1k
20 votes
Accepted

Variational Autoencoder − Dimension of the latent space

You seem to have misunderstood your architecture and are, quite simply, overfitting your data. It looks like your interpretation of the latent space is that it represents a manifold of realistic-...
jkm's user avatar
  • 2,165
20 votes

Balancing Reconstruction vs KL Loss Variational Autoencoder

Little late to the party here and you're probably way past this, but it's well documented you have to "warm up" the KL loss term by starting at zero and training a bit on just reconstruction loss ...
JPJ's user avatar
  • 1,461
19 votes

How to weight KLD loss vs reconstruction loss in variational auto-encoder?

I would like to add one more paper relating to this issue (I cannot comment due to my low reputation at the moment). In subsection 3.1 of the paper, the authors specified that they failed to train a ...
Cuong's user avatar
  • 451
17 votes
Accepted

How do Variational Auto Encoders backprop past the sampling step

The reparameterization trick. $$x = \text{sample}(\mathcal{N}(\mu, \sigma^2))$$ is not backpropable wrt $\mu$ or $\sigma$. However, we can rewrite this as: $$x = \mu + \sigma\ \text{sample}( \...
shimao's user avatar
  • 26.3k
16 votes

Pre-training in deep convolutional neural network?

As can be understood from the above answers, pre-training was 'fashioned out' when multiple things happened. However, I do want to distill my understanding of it: Long time ago in 2010, everyone ...
rhadar's user avatar
  • 990
16 votes
Accepted

When generating samples using variational autoencoder, we decode samples from $N(0,1)$ instead of $\mu + \sigma N(0,1)$

During training, we are drawing $z \sim P(z|x)$, and then decoding with $\hat x = g(z)$. During generation, we are drawing $z \sim P(z)$, and then decoding $x = g(z)$. So this answers your question:...
shimao's user avatar
  • 26.3k
13 votes

How does the reparameterization trick for VAEs work and why is it important?

I thought the explanation found in Stanford CS228 course on probabilistic graphical models was very good. It can be found here: https://ermongroup.github.io/cs228-notes/extras/vae/ I've summarized/...
horace he's user avatar
  • 131
13 votes

When should I use a variational autoencoder as opposed to an autoencoder?

TenaliRaman had some good points but he missed a lot of fundamental concepts as well. First it should be noted that the primary reason to use an AE-like framework is the latent space that allows us ...
JPJ's user avatar
  • 1,461
12 votes
Accepted

Is the output of a variational autoencoder meant to be a distribution that can be sampled, or a sample directly?

Question 1: The output of the decoder aims to model the distribution $p(x|t)$, i.e. the distribution of data $x$ given the latent variable $t$. Therefore, in principle, it should always be ...
learner's user avatar
  • 697
11 votes
Accepted

Variational autoencoder: Why reconstruction term is same to square loss?

For regular Autoencoders, you start from an input, $x$ and encode it to obtain your latent variable (or code), $z$, using some function that satisfy: $z=f(x)$. After getting the latent variable, you ...
Oringa's user avatar
  • 226
11 votes
Accepted

Why binary crossentropy can be used as the loss function in autoencoders?

I thought a regression loss function such as mean squared error or mean absolute error must be used instead, which have a value of zero when labels and predictions are the same. That's exactly ...
today's user avatar
  • 806
10 votes
Accepted

Why would I ever use a linear autoencoder for dimensionality reduction?

Using a linear autoencoder instead of PCA could also be useful in a large-scale learning scenario. Since you can use Stochastic Gradient Descent (SGD) to train the AE, there is no neeed to load all ...
Daniel López's user avatar
10 votes

What're the differences between PCA and autoencoder?

The general answer is that auto-associative neural networks can perform non-linear dimensionality reduction. Training the network is generally not as fast as PCA, so the trade-off is computational ...
Oliver Schulte's user avatar
10 votes
Accepted

Is Greedy Layer-Wise Training of Deep Networks necessary for successfully training or is stochastic gradient descent enough?

Pre-training is no longer necessary. Its purpose was to find a good initialization for the network weights in order to facilitate convergence when a high number of layers were employed. Nowadays, we ...
rcpinto's user avatar
  • 1,703
10 votes
Accepted

What is the origin of the autoencoder neural networks?

According to the history provided in Schmidhuber, "Deep learning in neural networks: an overview," Neural Networks (2015), auto-encoders were proposed as a method for unsupervised pre-training in ...
Sycorax's user avatar
  • 92.2k

Only top scored, non community-wiki answers of a minimum length are eligible