16
$\begingroup$

Recently, I have been studying autoencoders. If I understood correctly, an autoencoder is a neural network where the input layer is identical to the output layer. So, the neural network tries to predict the output using the input as golden standard.

What is the usefulness of this model? What are the benefits of trying to reconstruct some output elements, making them as equal as possible to the input elements? Why should one use all this machinery to get to the same starting point?

$\endgroup$
16
$\begingroup$

Auto encoders have an input layer, hidden layer, and an output layer. The input is forced to be as identical to the output, so its the hidden layer we are interested in.

The hidden layer form a kind of encoding of the input. "The aim of an auto-encoder is to learn a compressed, distributed representation (encoding) for a set of data." If input is a 100 dimensional vector, and you have 60 neurons in the hidden layer, then the auto encoder algorithm will replicate the input as a 100 dimensional vector in the output layer, in the process giving you a 60 dimensional vector that encodes your input.

So the purpose of auto encoders is dimensionality reduction, amongst many others.

$\endgroup$
  • $\begingroup$ Thanks @Madhulika, maybe I've understood. The algorithm works as follows: it has an input layer, it trains the neural network in order to have an output layer identical to the input layer. Than it compares the input layer with the output layer, and if they are different, it keeps on re-train the neural network. It stops when they are identical. When it finishes, we take the last hidden layer as the best dimensionality reduction approximation of the input layer, and use it for any goal we need. Is this correct? $\endgroup$ – larry Jan 16 '14 at 17:10
  • $\begingroup$ Yup, you got that nearly right. :) Read some further literature on it. $\endgroup$ – Madhulika Mukherjee Jan 23 '14 at 9:35
4
$\begingroup$

It can also model your population so that when you input a new vector, you can check how different is the output from the input. If they're "quite" the same, you can assume the input matches the population. If they're "quite" different, then the input probably doesn't belong to the population you modeled.

I see it as a kind of "regression by neural networks" where you try to have a function describing your data: its output is the same as the input.

$\endgroup$
3
$\begingroup$

Maybe these pictures give you some intuition. As commenter above said auto encoders tries to extract some high level features from the training examples. You may see how prognostication algorithm is used to train each hidden level separately for the deep NN on the second picture.

enter image description here

enter image description here

Pictures are taken from Russian wikipedia.

$\endgroup$
  • $\begingroup$ Comment to the pictures would be helpful. $\endgroup$ – Tim Mar 23 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.