Denoising Autoencoders (DAE) works by inducing some noise in the input vector and then transforming it into the hidden layer, while trying to reconstruct the original vector. However, I fail to understand the intuition of Contractive Autoencoders (CAE).
Does the CAE say that the hidden layer will only learn the features that differentiate one input from the rest of the inputs in a vector? If so then how would we reconstruct the original vector?
No. The CAE tries to make the encoder (i.e. mapping from input to hidden layer) have the property of locality, i.e. small changes in input lead to small changes at hidden layer. This is a nice property because it means the mapping is not too sensitive, which should help it generalise beyond the training data.
(There is an extra complication: the CAE tries in particular to enforce locality along the direction of the low-dimensional manifold which all autoencoders assume is present in the input data.
My understanding of this is really based on the original paper (Rifai et al.). This video explains the directionality a bit differently.
I find this part a bit harder to explain. I suggest to concentrate on the locality in the first place.)
