Discrete Random Variables and Deep Generative Models - Why Gumbel-Softmax is needed?

Question

I am reading this 2014 NIPS paper on deep generative models and their application to latent discrete random variables, and this 2017 ICLR paper on Gumbel-Softmax. I essentially don't understand why we needed the Gumbel reparametrization trick, if the earlier paper in 2014 is already using discrete random variables within the variational framework (without using Gumbel reparameterization). I know why we use the reparameterization trick in the standard variational auto-encoder. But, it seems the 2014 is just solving the same task as the 2017 paper but without the need for reparameterization.

Answer 1

The paper 2014 paper by Kingma et al does not deal with modelling of discrete latent variables per say. In this paper, $y$ represents the discrete label information. When we don't have a label, we instead estimate it using $q_{\phi}(y|x)$ (a classifier). However, this is has no probabilistic interpretation, as we cannot sample from it or score a sample.

To make learning tractable in this model, we instead resort to marginalisation as shown below: $$p(x) = \sum_{y} p(x, y)$$

You can imagine, as $y$ becomes high-dimensional, marginalisation becomes difficult. Now in the paper by Jang et al, $y$ is a true latent random variable. Therefore we can both sample $y \sim q_{\phi}(y|x)$ as well as score it. The main contribution of this paper (and this same one) is a reparametrisation trick of a relaxed one-hot represented categorical distribution $p(p,\tau)$ that converges to a categorical distribution $\text{Cat}(p)$ as $\tau \to 0$ is annealed.