I am reading this 2014 NIPS paper on deep generative models and their application to latent discrete random variables, and this 2017 ICLR paper on Gumbel-Softmax. I essentially don't understand why we needed the Gumbel reparametrization trick, if the earlier paper in 2014 is already using discrete random variables within the variational framework (without using Gumbel reparameterization). I know why we use the reparameterization trick in the standard variational auto-encoder. But, it seems the 2014 is just solving the same task as the 2017 paper but without the need for reparameterization.


1 Answer 1


The paper 2014 paper by Kingma et al does not deal with modelling of discrete latent variables per say. In this paper, $y$ represents the discrete label information. When we don't have a label, we instead estimate it using $q_{\phi}(y|x)$ (a classifier). However, this is has no probabilistic interpretation, as we cannot sample from it or score a sample.

To make learning tractable in this model, we instead resort to marginalisation as shown below: $$p(x) = \sum_{y} p(x, y)$$

You can imagine, as $y$ becomes high-dimensional, marginalisation becomes difficult. Now in the paper by Jang et al, $y$ is a true latent random variable. Therefore we can both sample $y \sim q_{\phi}(y|x)$ as well as score it. The main contribution of this paper (and this same one) is a reparametrisation trick of a relaxed one-hot represented categorical distribution $p(p,\tau)$ that converges to a categorical distribution $\text{Cat}(p)$ as $\tau \to 0$ is annealed.

  • $\begingroup$ Does this mean, using the framework of 2014, we can't do fully unsupervised setting, but the 2017 papers allow us to do this in the fully unsupervised setting? $\endgroup$ Oct 15, 2018 at 9:37
  • $\begingroup$ Also, it seems in equation 7 we sum over all possible values of $y$, but if we had a classifier in hand (trained using the labelled data), why wouldn't we just first predict the value of $y$ using the classifier, and then just make equation 7 similar to equation 6? The only difference is, in one case we had the label $y$, in the other case we used the classifier to predict it first. $\endgroup$ Oct 15, 2018 at 9:51
  • $\begingroup$ If we only use the unsupervised bound in the 2014 the contribution from $q_{\phi}(y|x)$ stays constant, therefore it has no effect on training. If we had a trained classifier it would still not be a latent variable model, because that classifier is deterministic. $\endgroup$
    – wohlert
    Oct 15, 2018 at 18:04
  • $\begingroup$ If we take equation 7 of the 2014 paper, what exactly changes in the math between this work and the 2017 papers (ignore the reparameterization for now)? Instead of taking the sum over all possible values of $y$, we sample from $q(y|x)$? Is that it? $\endgroup$ Oct 15, 2018 at 21:04
  • $\begingroup$ Yes, we just sample. $\endgroup$
    – wohlert
    Oct 15, 2018 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.