4
$\begingroup$

I'm trying to get a basic understanding of AutoEncoders. Basically they are neural networks with a very low representation of the original input in some hidden layer, and then a final layer which has exact same dimension as the input but 'reconstructed' from the latent lower-dimensional representation. What I don't understand is what should we do with autoencoders, apart from denoising or compression. For example, would it be possible to use the latent representation as a dimensionality-reducted version of the original input, and then using it to train some Machine Learning Model?

Many thanks,

James

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ As you mentioned in the question, Autoencoders serve as models which can reduce the dimensionality of their inputs. They are trained to "mimic" their inputs. The encoder produces a latent representation of the inputs which are passed to the decoder, where the inputs are reconstructed. Apart from reducing dimensionality, they are used to denoise images, colorize grayscale images and also in image segmentation ( UNet is such a model, which uses skip-connections too ). $\endgroup$ – Shubham Panchal Jun 22 at 14:22
7
$\begingroup$

Yes, dimension reduction is one way to use auto-encoders. Consider a feed-forward fully-connected auto-encoder with and input layer, 1 hidden layer with $k$ units, 1 output layer and all linear activation functions. The latent space of this auto-encoder spans the first $k$ principle components of the original data. This can be useful if you want to represent the input in fewer features, but don't necessarily care about the orthogonality constraint in PCA. (More information: What're the differences between PCA and autoencoder?)

But auto-encoders allow a number of variations on this basic theme, giving you more options for how the latent space should be constructed than does PCA.

  • Using CNN layers instead of FFNs is clearly a different kind of model compared to PCA, so it will encode a different kind of information in the latent space.
  • Using nonlinear activation functions will also yield a different kind of latent encoding than PCA (because PCA is linear).
  • Likewise sparse, contractive or variational auto-encoders have different goals than PCA and will give different results, which can be helpful depending on what problem you're trying to solve.
Improve this answer
$\endgroup$
2
  • $\begingroup$ The following is an (attempt!) of a practical use of the first paragraph. You would first run PCA and calculate a $k$ that explains a good amount of variance for the problem at hand (and also take any insights from the PCA applied to tour data). The difference between the above PCA $k$ and whatever the $k’$ is from fitting autoencoder may give an idea of the size difference in latent space (although probably difficult to interpret). $\endgroup$ – Single Malt Jun 22 at 18:57
  • 1
    $\begingroup$ @SingleMalt Not quite. The auto-encoder rank-$k$ basis and the PCA rank-$k$ basis both span the same basis. The PCA solution is constrained to be an orthogonal basis, while the auto-encoder’s is not. Choosing a different $k$ between the two is not a fair comparison, because the solution using larger $k$ has more basis vectors, potentially reducing the reconstruction error compared to using a smaller number. $\endgroup$ – Sycorax♦ Jun 22 at 19:11
2
$\begingroup$

Yes, using some form of autoencoder training/pre-training to create good features has been a successful approach in many areas. E.g. for tabular data, using a denoising autoencoder was the winning approach in a recent Kaggle competition. Autoencoder pre-training (recovering masked features) has been used in the TabNet paper that gained a decent amount of attention. These first two examples were for tabular data, but similar things have long been done (and are still popular) for vision applications as discussed here. However, as you may notice it looks like a lot of the most successful version of autoencoders for pretraining representations have been ones that were trained to re-construct corrupted inputs into the uncorrupted inputs.

Other ideas for what you can use autoencoders for would be file compression (e.g. for video calls, what's minimal information that still produces a video that looks decent to humans, but does not take a lot of bandwidth to transmit?).

Improve this answer
$\endgroup$
1
$\begingroup$

As you mentioned, auto-encoders can be used for dimensionality reduction. One of the nice things about them, is they are an unsupervised learning method. If you have a large volume of unlabeled data and a small volume of labeled data, you can train your auto-encoder on the large unlabeled dataset to get a robust representation of your data and then either train something lighter weight on your embedings or use transfer learning on your encoder.

There are also a number of other applications of auto-encoders though. For example, auto-encoders can be used for anomaly detection. Since the auto-encoder can encode more of the dataset correctly by robustly representing common patterns than unusual ones, the reconstruction error of "normal" data tends to be lower than that of "anomalies". The reconstruction error can then be used to identify anomalous data.

Another application for them can be grouping or clustering similar data. Data with similar properties will tend to have similar embedings, so two inputs with embedings that are similar are likely to have similar content. You can then apply traditional clustering methods to the pairwise embeding distances to find structure in your dataset or identify similar instances as in recommender systems. The downside is that the "similarity" between the inputs is not necessarily related to features that are important to a user (images of two clothing items may be similar because they are both red rather than because they are both dresses). If particular aspects of similarity are important, you may be better off using triplet loss or a Siamese network. However, both of those methods require some additional label information, where as once again, auto-encoders are unsupervised so they are an option even when no labels are available, or "similarity" is hard to define.

I'm sure there are plenty of other applications, these are just the ones I've used them for. That is one of the fun things about unsupervised methods; you can be creative with them without getting stuck waiting for annotations or cleaning bad labels.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.