What concept comes before VAE and after GMM? Suppose I am designing a course on generative models and I have just finished discussing GMM. My goal is to teach VAE.
However, VAE's technicality is very high. Does there exist some model in between GMM and VAE that's useful to as a conceptual bridge?
Sorry if this is a general question, but it is similar to how you wouldn't want the first lecture on supervised learning to be CNN, but rather build it up from linear regression or something.
 A: I'm assuming you've already explained that the goal of generative modelling is to estimate the probability distribution of the data $p(\mathbf{x})$. This is generally done in three steps:

*

*Propose a probability distribution $q(\mathbf{x};\theta)$ that will be used to approximate the true distribution $p(\mathbf{x})$ by estimating the parameters $\theta$ given the observed data.

*Choose a method to estimate $\theta$, such as maximum likelihood estimation.

*Estimate the parameters $\theta$.

The only difference between VAE's, GMM's, autoencoders, and really any other model occurs in step 1, since the distribution $q(\mathbf{x};\theta)$ is different in all of these.
I personally think the step between GMM's and VAE's is too big when trying to implement them in practice, but theoretically, they just propose two different $q$ distributions, so this could be the connection between the two concepts. You can find a list of generative models here, but again the only difference between these are the choice of $q$, and consequently the choice of conditional and unconditional independencies between random variables in the model.
A: Just simple AE (Autoencoders) are neural nets that do Identity function: $f(X)=X$. They are simpler than the VAE, but very interesting as you may experiment with LV (latent variables).
They have encoding/decoding functions and a loss function.
GMM? Well you think of them as generalizing k-means clustering procedure where we can detect the centers of the latent Gaussians (latent variables in general). So there are similarities with AE/VAE and any other generative models model in general since these generative models are made to learn the distribution of data classes (features).
All these are getting the information about the covariance structure of the data.
