9
$\begingroup$

As far as I know, both autoencoders and t-SNE are used for nonlinear dimensionality reduction. What are the differences between them and why should I use one versus another?

$\endgroup$
6
$\begingroup$

Both of them try to find a lower dimensionality embedding of your data. However, there are different minimization problems. More specifically, an autoencoder tries to minimize the reconstruction error, while t-SNE tries to find a lower dimensional space and at the same time it tries to preserve the neighborhood distances. As a result of this attribute, t-SNE is usually preferred for plots and visualizations.

$\endgroup$
  • $\begingroup$ So in this sense, does it mean autoencoder is better to find lower dimension when the lower dimension is more than 3D? (Because I assume if the lower dimension is large, t-SNE may not work that well?) $\endgroup$ – RockTheStar May 23 '15 at 23:58
  • 2
    $\begingroup$ There are many types of autoencoders (sparse, variational, stacked, convolutional etc.) depending on your task. They can be very efficient in discovering lower dimensional embeddings, based on reconstruction error. Therefore, if your task is to find an optimal lower dimensional space (even 2D) I would suggest you to pick the right autoencoder for your task. If you have to do visualisations then t-SNE would probably be your choice. as the neighbouring distances preservation, can result better visualisations. $\endgroup$ – Yannis Assael May 24 '15 at 1:05
1
$\begingroup$

[Autoencoders] primarily focus on maximizing the variance of the data in the latent space, as a result of which autoencoders are less successful in retaining the local structure of the data in the latent space than manifold learners...

From "Learning a Parametric Embedding by Preserving Local Structure", Laurens van der Maaten (https://lvdmaaten.github.io/publications/papers/AISTATS_2009.pdf)

$\endgroup$
  • $\begingroup$ This is not "the t-SNE paper". Standard t-SNE is non-parametric. $\endgroup$ – amoeba Jul 1 at 8:14
  • 1
    $\begingroup$ Thanks @amoeba for your comment on this. I edited the post. $\endgroup$ – User128525 Jul 1 at 8:24
0
$\begingroup$

Autoencoder and t-SNE can be used together for better visualization in high dimensional data, as described in [1]:

For 2D visualization specifically, t-SNE is probably the best algorithm around, but it typically requires relatively low-dimensional data. So a good strategy for visualizing similarity relationships in high-dimensional data is to start by using an autoencoder to compress your data into a low-dimensional space (e.g. 32 dimensional), then use t-SNE for mapping the compressed data to a 2D plane.


[1] https://blog.keras.io/building-autoencoders-in-keras.html

$\endgroup$
0
$\begingroup$

Autoencoder is designed to preserve previous data in a 2-norm sense, which can be thought as preserve the kinetic energy of the data, if data is velocity.

While t-SNE, use KL divergence which is not symmetrical, it will lead to t-SNE focus more on local structure, while autoencoder tends to keep overall L2 error small, which is in a global sense.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.