0
$\begingroup$

Both VAE and LDA (latent Dirichlet allocation) is based on variational inference, and they both try to optimize ELBO objective function

Variational autoencoders use reparameterization so that "backprop can be applied without being blocked by sampling."

In LDA (latent Dirichlet allocation), the model also sample a hidden variable (z) and use this z as parameter to generate other observed variable (word).

LDA can be solved by a nice/clear variational inference (coordinate ascent mean-field algorithm), without any "reparameterization trick". Then why "reparameterization trick" is a must-have for VAE?

Improve this question
$\endgroup$
1
  • $\begingroup$ LDA is Latent Dirichlet allocation $\endgroup$ – user3462510 Sep 25 '20 at 4:07
1
$\begingroup$

The distinction between VAE and LDA is that LDA doesn't use back-propagation for inference, while VAE does.

LDA inference can use Monte Carlo, Variational Bayes, or certain likelihood maximization procedures can be used instead.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ More to the point, VAEs perform variational inference using backpropagation, while the structure of LDA allows a coordinate ascent approach to variational inference that does not rely on backpropagation. $\endgroup$ – jkpate Sep 25 '20 at 14:57
  • $\begingroup$ @jkpate It sounds like you have some expertise in this area. Perhaps you could add an answer which develops this comment in more detail. $\endgroup$ – Sycorax♦ Sep 25 '20 at 15:15
  • 1
    $\begingroup$ Thanks, I think I also found some related answers. ("backprop" is kind of general word. the coordinate ascent mean-field algorithm used for LDA is also based on gradient-oriented iterations,similar to backprop. so I think "LDA did not not use backprop" did not hit the pain point of the question. The key is "why LDA can be solved without using backprop and reparameterization-trick, while VAE has to"? $\endgroup$ – user3462510 Sep 26 '20 at 0:18
  • $\begingroup$ My understanding is: in VAE, we use one sample of Z to approximate E_{q(z)} P(x|z). while in LDA for a similar term E_{q(𝑧,𝛽)} P(𝐸𝑞[𝑙𝑜𝑔𝑝(𝑤|𝑧,𝛽)]), LDA does not try to use one sample of z to approximate this term. in LDA, z is a discrete variable with multinomial dist. z only has a few hundred possible values, So LDA can easily go through all z values and integrate it out. in VAE, this z is continuous vector, and VAE does not have a way to integrate z out, so VAE has to rely on z-value, and thus VAE has to rely on reparameterization-trick to represent z. This is my 2 cents $\endgroup$ – user3462510 Sep 26 '20 at 0:30
  • $\begingroup$ @user3462510 It seems like you've given this some thought. I think these additions would make a good answer. $\endgroup$ – Sycorax♦ Sep 26 '20 at 0:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.