Why Reparameterization Trick does not work with discrete latent variables?

I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete latent variables. Reparameterization changes z as $$\mu$$+$$\epsilon$$*$$\sigma$$. We can determine mean and standard deviation of discrete variables too. So, why does the Reparameterization trick not work with them?

Could anyone please provide me insight on this?

• Not sure if you understood what reparameterization actually achieves. You may want to see how the distribution is defined and how backprop is decoupled from sampling with reparameterization and why that works. Commented Apr 21, 2023 at 8:57

The re-parameterization $$x=\mu + \epsilon \sigma$$ where $$\mu, \sigma>0$$ are the outputs of a neural network and $$\epsilon \sim \mathcal N(0,1^2)$$ is a standard normal variable implies that $$x\sim \mathcal N(\mu,\sigma^2)$$, i.e. $$x$$ has normal distribution with mean $$\mu$$ and standard deviation $$\sigma$$. Therefore, $$x$$ is not a discrete random variable.

If you have a different reparameterization in mind, or a different sampling of $$\epsilon$$, then that would change this result, but it's not clear if that's what you're asking.

• You seem to imply that the trick is only about Gaussians rather than continuous variables, what would mean that it doesn’t hold for all continuous variables, just for Gaussians.
– Tim
Commented May 29, 2022 at 18:01
• @Tim I've edited the answer Commented May 29, 2022 at 20:49
• Thank you for your answer @user336650. Yes, "after" reparameterization, x follows normal distribution with mean 𝜇 and standard deviation 𝜎. However, suppose our input was discrete, like black and white image. In that case, for that discrete input, we can have still have its mean and standard deviation, which means, we can still perform the equation 𝜇+𝜖𝜎 in it, isn't it? So, my question is, why it is mentioned by the professor in the youtube link in my question description that reparameterization (which I understand as : 𝜇+𝜖𝜎 ) cannot be applied to discrete variables? Commented May 30, 2022 at 8:18
• @Curious $\mu,\sigma$ are outputs of a neural network, so we're not using the input (b&w image) to compute $x$ directly. Even if we accept your reasoning, consider a Bernoulli r.v. It has mean $p$ and variance $p (1 - p)$ but using $p + \epsilon\sqrt{p(1-p)}$ is not a sample from a Bernoulli distribution. Commented May 31, 2022 at 13:55
• Yup, 𝑝 + 𝜖 √[p(1-p)] is not a sample from Bernoulli Distr. It is sample from Normal Distr, right? But still we are being able to apply reparameterization trick mean+𝜖*stdv in Bernoulli r.v. too, right? If yes, that is why I am not being able to understand why Reparameterization Trick does not work with discrete latent variables. @user336650 Commented May 31, 2022 at 15:55

In autoencoding variational bayes and other variational methods, the reparametrization trick is used to sample from an approximate posterior in a way that is amenable to gradient based optimization.

Sampling directly from $$p_\phi(z|x)= \mathcal N(\mu(x), \sigma(x))$$ requires more complex automatic differentiation, while sampling like $$z= \mathcal \mu(x) + \sigma(x) \cdot \epsilon$$, $$\epsilon \sim N(\mathbf 0, \mathbb I)$$, uses only operations that can be easily differentiated.

Discrete latent variable, on the other hand, even if defined taking subderivatives into account, would lead to zero derivatives, thus making it impossible to optimise through them.

There is an approximate solution to that, however, in the form of the Straight-Through estimators, which basically treat the variables as continuous for the backward pass. See for example the Straight-Through Gumbel Softmax.