Understanding reparameterization trick and training process in variational autoencoders

Question

I am trying to understand variational autoencoders, particularly the sampling component and the reparameterization trick. I understand that instead of using a fixed determinstic latent representation as in traditional autoencoders, variational autoencoders involve computing mean and standard deviation vectors. These vectors are then used to sample latent vectors, which in turn can generate new data.

However, I am trying to understand where sampling fits in the training process, if at all? My understanding is that the stochastic aspect is only relevant for generating new data (after training is complete), but not for training the encoder/decoder networks. Is that correct? Or does sampling also occur during training the variational autoencoder model, and if so how does this work?

Answer 1

Sampling is used during VAE training and during inference. The idea is that the data are encoded as random draws from a particular distribution. That distribution's parameters depend on the input data, because they are determined from the inputs. By contrast, an ordinary autoencoder maps the data to a vector deterministically, without any sampling.

For a Gaussian VAE, each input is encoded to 2 vectors, one for means and one for standard deviations. Suppose your latent representation is 2 dimensional. You have a mean vector $\mu = \begin{bmatrix}\mu_1 \\ \mu_2 \end{bmatrix}$ and covariance $\Sigma = \begin{bmatrix}\sigma_1^2 & 0 \\ 0 & \sigma_2^2 \end{bmatrix}$. Then you use the re-parameterization trick to draw a random vector from this distribution. Then the decoder reconstructs the data.

A more expansive discussion can be found here: What are variational autoencoders and to what learning tasks are they used?