To my knowledge, Autoencoders are an unsupervised learning technique in which we leverage neural networks for the task of representation learning.
I know there is a lot of topics around autoencoders, I can find the code and run but I'm unable to understand the math and concept of Autoencoders.
I'm looking for intuition in words but also some simple insight into mathematical calculations (I don't know if the latter is possible). Can anybody shed some light on the "AutoEncoders"?
Basically, you reduce the input (dimension) by reducing the neurons per layer. This is the encoder part. Afterwards, based on that very low dimensional representation, the entire data is broken down to only important features. This is what you finally wanna' have: A meaningful, low dimensional data representation. To train and validate this encoder function, you decode this data by mostly using the same topology in a reversed manner. Now there are two options:
A good example, e.g., is the use for mobile communication which breaks down/encodes the data so that only a significant part of your speech remains, then this data is transfered and at the receiver side it is decoded. Despite from that concept the handling and so on is quite the same like for every neural network (train, validate, minimize error function and so on..).
Autoencoders are for finding a (typically) lower dimensional representation of your data. In other words, it tries to compress/encode your data, which the name comes from.
There are mainly two components: encoder and decoder. Encoder's responsibility is to compress your data (maps it onto another vector space), and Decoder's responsibility is get back the original point, given the encoded version.
Both encoder and decoder are neural networks. The aim is to minimize the reconstruction error, i.e. an input $\mathbf x$ is given to the encoder which outputs $\mathbf y$, and then $\mathbf y$ is given to the decoder which outputs $\mathbf x'$, an estimate of the original data point, $\mathbf x$. The error to be minimized is the reconstruction error $||\mathbf x-\mathbf x'||^2$.
So, just like any other neural network, there is an input/output relationship, target variable, and a cost function. The rest is back-propagation.
Note that this is the common type of an autoencoder, and there are variations and additions in the literature.
I think it's useful to understand Autoencoders by first considering a linear autoencoder (ie a single linear layer for each of encoding and decoding networks).
Minimising the reconstruction error using n bottleneck nodes (with input of N>n) then corresponds to projecting to the subspace spanned by the first n principal components (see dimensionality reduction section of PCA Wiki).
So if you are familiar with principal components analysis this should give you an insight on how and when they are useful.
regular autoencoders allow to identify nonlinear relationships too so eg if your typical data is 2 dimensional of the form $(x, x^2)$ + noise then the autoencoder can extract the 'x' from both inputs in a single bottleneck node and then reconstruct $(x,x^2)$
