In a recent assignment, we were told to use PCA on the MNIST digits to reduce the dimensions from 64 (8 x 8 images) to 2. We then had to cluster the digits using a Gaussian Mixture Model. PCA using only 2 principal components does not yield distinct clusters and as a result the model is not able to produce useful groupings.

However, using t-SNE with 2 components, the clusters are much better separated. The Gaussian Mixture Model produces more distinct clusters when applied to the t-SNE components.

The difference in PCA with 2 components and t-SNE with 2 components can be seen in the following pair of images where the transformations have been applied to the MNIST dataset.



I have read that t-SNE is only used for visualization of high dimensional data, such as in this answer, yet given the distinct clusters it produces, why is it not used as a dimensionality reduction technique that is then used for classification models or as a standalone clustering method?

  • 2
    $\begingroup$ Do you mean classification or clustering? The title says clustering but the post says classification. $\endgroup$ – usεr11852 Apr 12 '18 at 18:29
  • $\begingroup$ Sorry about that. I want to know why it is not used as a clustering technique or as a dimensionality reduction technique for classification. I've edited to reflect this. $\endgroup$ – willk Apr 12 '18 at 19:18
  • $\begingroup$ Coincidentally enough, a paper released recently uses t-SNE and an unsupervised clustering algorithm to label combustion processes. $\endgroup$ – tpg2114 Apr 12 '18 at 20:01
  • 2
    $\begingroup$ The answer that you linked demonstrates how misleading tSNE can be. You see clusters in the plot that do not exist in the data. That is harmful if you don't have labels. And don't draw too many conclusions from MNIST data. That is an extremely well behaved data set... $\endgroup$ – Has QUIT--Anony-Mousse Apr 12 '18 at 21:00
  • 2
    $\begingroup$ I've found this article to be helpful in explaining t-SNE and its drawbacks. It has plenty of interactive visualizations that help emphasize the main points. $\endgroup$ – willk Apr 13 '18 at 1:04

The main reason that $t$-SNE is not used in classification models is that it does not learn a function from the original space to the new (lower) dimensional one. As such, when we would try to use our classifier on new / unseen data we will not be able to map / pre-process these new data according to the previous $t$-SNE results.

There is work on training a deep neural network to approximate $t$-SNE results (e.g., the "parametric" $t$-SNE paper) but this work has been superseded in part by the existence of (deep) autoencoders. Autoencoders are starting to be used as input / pre-processors to classifiers (especially DNN) exactly because they get very good performance in training as well as generalise naturally to new data.

$t$-SNE can be potentially used if we use a non-distance based clustering techniques like FMM (Finite Mixture Models) or DBSCAN (Density-based Models). As you correctly note, in such cases, the $t$-SNE output can quite helpful. The issue in these use cases is that some people might try to read into the cluster placement and not only the cluster membership. As the global distances are lost, drawing conclusions from cluster placement can lead to bogus insights. Notice that just saying: "hey, we found all the 1s cluster together" does not offer great value if cannot say what they are far from. If we just wanted to find the 1's we might as well have used classification to begin with (which bring us back to the use of autoencoders).

  • 1
    $\begingroup$ The Q seems to ask more about clustering than about classification. At least clustering is in the title. $\endgroup$ – amoeba Apr 12 '18 at 18:25
  • $\begingroup$ @amoeba: I thought the same and wrote about potential use through non-distance based clustering (eg. FMM, DBSCAN), but then I read the question: "why is it not used as a dimensionality reduction technique that is then used for classification models?" $\endgroup$ – usεr11852 Apr 12 '18 at 18:26
  • $\begingroup$ Yes, but the title Q is different. I think OP might be confused about the difference so it might make sense to address both in your A! $\endgroup$ – amoeba Apr 12 '18 at 18:29
  • 4
    $\begingroup$ OK.. OK... Slave-driving eukaryote... :P $\endgroup$ – usεr11852 Apr 12 '18 at 18:31
  • 1
    $\begingroup$ (+1) I'd be very interested in hearing your thoughts on this clustering/t-SNE answer stats.stackexchange.com/questions/263539 I just posted. CC also to @caseWestern - this might be of interest to you too. $\endgroup$ – amoeba Jun 19 '18 at 14:54

t-SNE does not preserve distances, but it basically estimates probability distributions. In theory, the t-SNE algorithms maps the input to a map space of 2 or 3 dimensions. The input space is assumed to be a Gaussian distribution and the map space a t-distribution. The loss function used is the KL Divergence between the two distributions which is minimized using gradient descent.

According to Laurens van der Maaten who is a co-author of t-SNE

t-SNE does not retain distances but probabilities, so measuring some error between the Euclidean distances in high-D and low-D is useless.





As a general statement: given a sufficiently powerful (/suitable) classifier, or cluster-er, one would never apply any dimensionality reduction.

Dimensionality reduction loses information.

Since such a cluster-er or classifier (esp classifiers, less so clusterers), internally incorperates some form of projection to a meaningful space already. And Dimensionality reduction is also projection to a (hopefuly) meaningful space.

But dimensionality reduction has to do so in a uninformed way -- it does not know what task you are reducing for. This is especially true for classification, where you have outright supervised information. But it also applies to clustering, where the space one would want to project to for clustering is better defined (for this algorithm) than just "have less dimensions). @usεr11852's answer talks about this. As I said dimensionality reduction does not know what task you are reducing for -- you inform it in your choice of which dimensionality reduction algorithm you to use.

So often rather than adding a dimensionality reduction step as preprocessing before clustering/classification, one is better to use a different classifier/cluster-er that incorperates a useful projection.

One thing dimentionality reduction does have going for it in this though is its unsupervised nature in creating the projection to the (hopefully) meaningful space. Which is useful if you have little label data. But there are often other methods that are closely linked to your classifier (e.g. for neural networks, using autoencoder e.g. deep belief network pretraining) that are going to work better, because they are designed with that final task in mind. Not the more general task of dimensionality reduction.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.