I have been trying to understand how hierarchical latent variable models are different from multilayered autoencoders and in specific the argument below

Autoencoder networks resemble in many ways single-layer latent variable models. The key idea is that the inference process of mapping observations $x(t)$ to the corresponding latent variables, now called hidden unit activations $h(t)$, is modeled by an encoder network $f$ and the mapping back to observations is modeled by a decoder network g:

$h(t) = f(x(t); ξf ) - (5)$

$x'(t) = g(h'(t); ξg) - (6)$

The mappings $f$ and $g$ are called encoder and decoder mappings, respectively. In connection to latent variable models, analogous mappings are called the recognition and reconstruction mappings. Learning of autoencoders is based on minimizing the difference between the observation vector $x(t)$ and its reconstruction $x'(t)$, that is, minimizing the cost $||x(t) − x'(t)||^2$ with respect to the parameteres $ξf$ and $ξg$. Just like latent variable models, autoencoders can be stacked together:

$h^{(l)}_{(t)} = f^{(l)}(h^{(l−1)}_{(t)}) (7)$

$h'^{(l−1)}_{(t)} = g^{(l)}(h'^{(l)}_{(t)}) (8)$

As before, the observations are taken into the equation by defining $h^0 := x$. Furthermore, now $h'^L := h^L$ for the last layer $L$, connecting the encoder and decoder paths. Typically over the course of learning, new layers are added to the previously trained network. After adding and training the last layer, training can continue in a supervised manner using just the mappings $f^l$ , which define a multi-layer feedforward network, and minimizing the squared distance between the actual outputs $h^L$ and desired targets outputs. It is tempting to assume that the hierarchical version of the autoencoder in Eqs. (7–8) corresponds somehow to the hierarchical latent variable model in Eq. (4). Unfortunately this is not the case because the intermediate hidden layers $0 < l < L$ act as so called deterministic variables while the hierarchical latent variable model requires so called stochastic variables. The difference is that stochastic variables have independent representational capacity. No matter what the priors tell, stochastic latent variables $s^l$ can overrule this and add their own bits of information to the reconstruction.By contrast, deterministic variables such as $h'^l$ add zero bits of information

How do latent variables manage adding their own bits of information and how can they do what autoencoders cannot?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.