57

After reading through Kingma's NIPS 2015 workshop slides, I realized that we need the reparameterization trick in order to backpropagate through a random node. Intuitively, in its original form, VAEs sample from a random node $z$ which is approximated by the parametric model $q(z \mid \phi, x)$ of the true posterior. Backprop cannot flow through a random ...


56

Assume we have a normal distribution $q$ that is parameterized by $\theta$, specifically $q_{\theta}(x) = N(\theta,1)$. We want to solve the below problem $$ \text{min}_{\theta} \quad E_q[x^2] $$ This is of course a rather silly problem and the optimal $\theta$ is obvious. However, here we just want to understand how the reparameterization trick helps in ...


37

Even though variational autoencoders (VAEs) are easy to implement and train, explaining them is not simple at all, because they blend concepts from Deep Learning and Variational Bayes, and the Deep Learning and Probabilistic Modeling communities use different terms for the same concepts. Thus when explaining VAEs you risk either concentrating on the ...


33

For a long answer, see Blei, Kucukelbir and McAuliffe here. This short answer draws heavily therefrom. MCMC is asymptotically exact; VI is not. In the limit, MCMC will exactly approximate the target distribution. VI comes without warranty. MCMC is computationally expensive. In general, VI is faster. Meaning, when we have computational time to kill and ...


20

Your approach is correct. EM is equivalent to VB under the constraint that the approximate posterior for $\Theta$ is constrained to be a point mass. (This is mentioned without proof on page 337 of Bayesian Data Analysis.) Let $\Theta^*$ be the unknown location of this point mass: $$ Q_\Theta(\Theta) = \delta(\Theta - \Theta^*) $$ VB will minimize the ...


17

A reasonable example of the mathematics of the "reparameterization trick" is given in goker's answer, but some motivation could be helpful. (I don't have permissions to comment on that answer; thus here is a separate answer.) In short, we want to compute some value $G_\theta$ of the form, $$G_\theta = \nabla_{\theta}E_{x\sim q_\theta}[\ldots]$$ Without the ...


16

In the context of computational problems, including numerical methods for Bayesian inference, the phrase "too expensive" generally could refer to two issues a particular problem is too "large" to compute for a particular "budget" a general approach scales badly, i.e. has high computational complexity For either case, the computational resources comprising ...


16

For anyone stumbling on this post also looking for an answer, this twitter thread has added a lot of very useful insight. Namely: beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework discusses my exact question with a few experiments. Interestingly, it seems their $\beta_{norm}$ (which is similar to my normalised KLD weight) ...


12

VAE is a framework that was proposed as a scalable way to do variational EM (or variational inference in general) on large datasets. Although it has an AE like structure, it serves a much larger purpose. Having said that, one can, of course, use VAEs to learn latent representations. VAEs are known to give representations with disentangled factors [1] This ...


10

Let me explain first, why do we need Reparameterization trick in VAE. VAE has encoder and decoder. Decoder randomly samples from true posterior Z~ q(z∣ϕ,x). To implement encoder and decoder as a neural network, you need to backpropogate through random sampling and that is the problem because backpropogation cannot flow through random node; to overcome this ...


10

Capacity is a loosely defined term, and there are several ways to measure it, some more rigorous than others. Roughly speaking, the capacity of a model describes how complex a relationship it can model. You could expect a model with higher capacity to be able to model more relationships between more variables than a model with a lower capacity. VC ...


10

The encoder distribution is $q(z|x)=\mathcal{N}(z|\mu(x),\Sigma(x))$ where $\Sigma=\text{diag}(\sigma_1,\ldots,\sigma_n)$, while the latent prior is given by $p(z)=\mathcal{N}(0,I)$. Both are multivariate Gaussians of dimension $n$, for which in general the KL divergence is: $$ \mathfrak{D}_\text{KL}[p_1\mid\mid p_2] = \frac{1}{2}\left[\log\frac{|\Sigma_2|}{|...


9

I thought the explanation found in Stanford CS228 course on probabilistic graphical models was very good. It can be found here: https://ermongroup.github.io/cs228-notes/extras/vae/ I've summarized/copied the important parts here for convenience/my own understanding (although I strongly recommend just checking out the original link). So, our problem is that ...


9

I will give you an example on discrete case to show why integration / sum over is very expensive. Suppose we have $100$ binary random variables, and we have the joint distribution $P(X_1, X_2, \cdots, X_{100})$. (In fact, it is impossible to store the joint distribution in a table, because there are $2^{100}$ values. Let us assume we have the it in table ...


9

To my best memory, I've never come across a formal definition for this in a statistical text, but I think you can stitch one together from a few contextual readings. Start with Bayesian Data Analysis, p. 261: Bayesian computation revolves around two steps: computation of the posterior distribution, $p(\theta|y)$, and computation of the posterior ...


8

Normal distribution is not the only distribution used for latent variables in VAEs. There are also works using von Mises-Fisher distribution (Hypershperical VAEs [1]), and there are VAEs using Gaussian mixtures, which is useful for unsupervised [2] and semi-supervised [3] tasks. Normal distribution has many nice properties, such as analytical evaluation of ...


7

I have a feeling that you treat $p$ as a completely unknown object. I do not think this is the case. This is probably what you missed. Say we observe $Y = \{y_i\}_{i=1}^n$ (i.i.d.) and we want to infer $p(x|Y)$ where we assume that $p(y|x)$ and $p(x)$ for $x\in\mathbb{R}^d$ are specified by the model. By Bayes' rule, $$p(x|Y) = \frac{p(x)}{p(Y)}p(Y|x) = \...


6

The MLE estimate $\hat{\alpha}$ could be used as location parameter of a Normal distribution with scale parameter $\sigma$ used as prior distribution of the scaling parameter. Then, such a prior distribution can be updated into a posterior via Metropolis-Hastings algorithm, i.e. a Markov Chain Monte Carlo method used to obtain a sequence of random samples ...


6

The KL divergences can be seen as a product of a weighting function $w(x)$ and a penalty function $g(x)$, i.e. $KL(q||p) = \sum_x w(x)g(x)$ with $w(x) = q(x)$ and $g(x) = \log\frac{q(x)}{p(x)}$ in the case of the reverse KL divergence. Whenever the weighting function is close to zero, i.e. $w(x) \approx 0$, the product of $w(x)g(x)$ is also close to zero and ...


6

Similar to Auto-encoders, the objective of a Variational Auto-encoder is to reconstruct the input. The only difference is that AEs have direct links between encoder and decoder parts, but VAEs have a sampling layer which samples form a distribution (usually a Gaussian) and then feeds the generated samples to the decoder part. Here are some examples from ...


6

Notice that by replacing $\sigma_1$ with $\sigma_1^2$ in the last equation you recover the previous (i.e. $\log(\sigma_1) - \sigma_1 \rightarrow 2\log(\sigma_1) - \sigma_1^2$). Leading me to think that in the first case the encoder is used to predict the variance, whereas in the second it is used to predict the standard deviation. Both formulations are ...


6

TenaliRaman had some good points but he missed a lot of fundamental concepts as well. First it should be noted that the primary reason to use an AE-like framework is the latent space that allows us to compress the information and hopefully get independent factors out of it that represent high-level features of the data. An important point is that, while ...


6

I would like to add one more paper relating to this issue (I cannot comment due to my low reputation at the moment). In subsection 3.1 of the paper, the authors specified that they failed to train a straight implementation of VAE that equally weighted the likelihood and the KL divergence. In their case, the KL loss was undesirably reduced to zero, although ...


6

Yes, it has been done. The following paper implements something of that form: Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders. Nat Dilokthanakul, Pedro A.M. Mediano, Marta Garnelo, Matthew C.H. Lee, Hugh Salimbeni, Kai Arulkumaran, Murray Shanahan. They experiment with using this approach for clustering. Each Gaussian in the ...


5

Taking the equation from Wikipedia $D_{KL}(Q||P) = \sum_\limits{z}Q(Z)\log\frac{Q(Z)}{P(Z,X)} +\log P(X)$ What we want is to minimize KL distance wrt $Q$ distribution. Since $P(X)$ is independent of $Q$ we need to care only about the first term. Substituting the factored approximation, $Q=\prod\limits_{i=1}^M q(Z_i|X)$ $$\sum_\limits{z}Q(Z)\log\frac{Q(...


5

Usually when performing Bayesian inference it's easy to encounter heavy integration over nuisance variables for instance. Another example can be a numerical sampling as in this case from a likelihood function, meaning to perform a random sampling from a given distribution. As the number of model parameters increases, this sampling becomes extremely heavy ...


5

For regular Autoencoders, you start from an input, $x$ and encode it to obtain your latent variable (or code), $z$, using some function that satisfy: $z=f(x)$. After getting the latent variable, you aim to reconstruct the input using some other function $\hat{x}=g(f(x))$. The reconstruction loss is yet another function $L(x,\hat{x})$ that you use to back-...


5

The variational Bayes approach to a Bayesian mixture model, as described in detail in the Wikipedia page, is producing a pseudo-posterior distribution on the parameters of the mixture model, including the latent variables $\mathbf{Z}$, by imposing a certain dependence structure (or graphical model) and estimating the hyperparameters of this model by a ...


4

We have our probablistic model. And want to recover parameters of the model. We reduce our task to optimizing variational lower bound (VLB). To do this we should be able make two things: calculate VLB get gradient of VLB Authors suggest using Monte Carlo Estimator for both. And actually they introduce this trick to get more precise Monte Carlo Gradient ...


Only top voted, non community-wiki answers of a minimum length are eligible