# Why aren't auto-encoders also considered generative models?

Auto-encoders (AEs) are composed of an encoder and a decoder (often represented by a neural network). The encoder produces a vector representation $$z$$ of its input $$x$$ (e.g. an image). The decoder attempts to output an object similar to $$x$$, $$x'$$, given $$z$$.

Variational auto-encoders (VAEs) are auto-encoders that model the hidden (or latent) space, which is the output space of the encoder, and the output space of the decoder as a probability distribution (e.g. a Gaussian). Hence, we can sample different $$z$$ from this hidden space, for each $$x$$, and, similarly, we can sample different $$x'$$ from the output space of the decoder (given the sampled $$z$$). For this reason, VAEs are considered generative models.

The difference between AEs and VAEs is that VAEs are considered generative models, whereas standard AEs are not. Why is that the case? Is it only because they are usually not formalised as generative models? More precisely, why are VAEs generative models (according to the definition of a generative model) and AEs are not? For example, a denoising AE can attempt to generate an original denoised image, so, in this sense, it is a generative model. Could AEs be formulated as generative models?

• In very short (not a proper answer), I would point out that we cannot sample for an AE. In a similar vain, an AE is a purely deterministic model, like a projection mechanism, while the VAE once again "generates" outputs. Jun 4, 2019 at 23:51

A generative model, as defined on the Wikipedia link you provided, aims to estimate the joint distribution of your data and some latent (random) variables usually $$p(\textbf{x},\textbf{z})$$. Particularly, in the case of the VAE you have that the data (usually $$\textbf{x}$$) are the images, text, audio or whatever you are modeling and the latent variable (usually $$\textbf{z}$$) is a multivariate normal (you can relax this). In the AE you cannot make this analogy, you have your data, you map it to a space which lies on a smaller dimension than your original image, and you try to decode this lower dimensional data into your original data. This means, there are no distributional assumptions of how your data is generated. In probabilistic reasoning lingo, there are no assumptions on the data generation process.
\begin{align} p(\textbf{x}) = p(x_{0}, x_{1},...,x_{N}) \\ &= \prod_{i=0}^{N}p(x_{i}|\textbf{x}_{i<}) \\ &= p(x_{N}|x_{N-1}, x_{N-2},...x_{0})p(x_{N-1}|x_{N-2}, x_{N-3},...x_{0})...p(x_{o}) \end{align}