# Why VAE need reparameterization trick while LDA does not (both using variational inference for optimization)

Both VAE and LDA (latent Dirichlet allocation) is based on variational inference, and they both try to optimize ELBO objective function

Variational autoencoders use reparameterization so that "backprop can be applied without being blocked by sampling."

In LDA (latent Dirichlet allocation), the model also sample a hidden variable (z) and use this z as parameter to generate other observed variable (word).

LDA can be solved by a nice/clear variational inference (coordinate ascent mean-field algorithm), without any "reparameterization trick". Then why "reparameterization trick" is a must-have for VAE?

• LDA is Latent Dirichlet allocation Sep 25, 2020 at 4:07

The distinction between VAE and LDA is that LDA doesn't use back-propagation for inference, while VAE does.

LDA inference can use Monte Carlo, Variational Bayes, or certain likelihood maximization procedures can be used instead.

• More to the point, VAEs perform variational inference using backpropagation, while the structure of LDA allows a coordinate ascent approach to variational inference that does not rely on backpropagation. Sep 25, 2020 at 14:57
• @jkpate It sounds like you have some expertise in this area. Perhaps you could add an answer which develops this comment in more detail.
– Sycorax
Sep 25, 2020 at 15:15
• Thanks, I think I also found some related answers. ("backprop" is kind of general word. the coordinate ascent mean-field algorithm used for LDA is also based on gradient-oriented iterations,similar to backprop. so I think "LDA did not not use backprop" did not hit the pain point of the question. The key is "why LDA can be solved without using backprop and reparameterization-trick, while VAE has to"? Sep 26, 2020 at 0:18
• My understanding is: in VAE, we use one sample of Z to approximate E_{q(z)} P(x|z). while in LDA for a similar term E_{q(𝑧,𝛽)} P(𝐸𝑞[𝑙𝑜𝑔𝑝(𝑤|𝑧,𝛽)]), LDA does not try to use one sample of z to approximate this term. in LDA, z is a discrete variable with multinomial dist. z only has a few hundred possible values, So LDA can easily go through all z values and integrate it out. in VAE, this z is continuous vector, and VAE does not have a way to integrate z out, so VAE has to rely on z-value, and thus VAE has to rely on reparameterization-trick to represent z. This is my 2 cents Sep 26, 2020 at 0:30
• @user3462510 It seems like you've given this some thought. I think these additions would make a good answer.
– Sycorax
Sep 26, 2020 at 0:41