Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?

Question

In a recent assignment, we were told to use PCA on the MNIST digits to reduce the dimensions from 64 (8 x 8 images) to 2. We then had to cluster the digits using a Gaussian Mixture Model. PCA using only 2 principal components does not yield distinct clusters and as a result the model is not able to produce useful groupings.

However, using t-SNE with 2 components, the clusters are much better separated. The Gaussian Mixture Model produces more distinct clusters when applied to the t-SNE components.

The difference in PCA with 2 components and t-SNE with 2 components can be seen in the following pair of images where the transformations have been applied to the MNIST dataset.

I have read that t-SNE is only used for visualization of high dimensional data, such as in this answer, yet given the distinct clusters it produces, why is it not used as a dimensionality reduction technique that is then used for classification models or as a standalone clustering method?

Answer 1

The main reason that $t$-SNE is not used in classification models is that it does not learn a function from the original space to the new (lower) dimensional one. As such, when we would try to use our classifier on new / unseen data we will not be able to map / pre-process these new data according to the previous $t$-SNE results.

There is work on training a deep neural network to approximate $t$-SNE results (e.g., the "parametric" $t$-SNE paper) but this work has been superseded in part by the existence of (deep) autoencoders. Autoencoders are starting to be used as input / pre-processors to classifiers (especially DNN) exactly because they get very good performance in training as well as generalise naturally to new data.

$t$-SNE can be potentially used if we use a non-distance based clustering techniques like FMM (Finite Mixture Models) or DBSCAN (Density-based Models). As you correctly note, in such cases, the $t$-SNE output can quite helpful. The issue in these use cases is that some people might try to read into the cluster placement and not only the cluster membership. As the global distances are lost, drawing conclusions from cluster placement can lead to bogus insights. Notice that just saying: "hey, we found all the 1s cluster together" does not offer great value if cannot say what they are far from. If we just wanted to find the 1's we might as well have used classification to begin with (which bring us back to the use of autoencoders).

Answer 2

t-SNE does not preserve distances, but it basically estimates probability distributions. In theory, the t-SNE algorithms maps the input to a map space of 2 or 3 dimensions. The input space is assumed to be a Gaussian distribution and the map space a t-distribution. The loss function used is the KL Divergence between the two distributions which is minimized using gradient descent.

According to Laurens van der Maaten who is a co-author of t-SNE

t-SNE does not retain distances but probabilities, so measuring some error between the Euclidean distances in high-D and low-D is useless.

Reference:

https://lvdmaaten.github.io/tsne/

https://www.oreilly.com/learning/an-illustrated-introduction-to-the-t-sne-algorithm

Answer 3

As a general statement: given a sufficiently powerful (/suitable) classifier, or cluster-er, one would never apply any dimensionality reduction.

Dimensionality reduction loses information.

Since such a cluster-er or classifier (esp classifiers, less so clusterers), internally incorperates some form of projection to a meaningful space already. And Dimensionality reduction is also projection to a (hopefuly) meaningful space.

But dimensionality reduction has to do so in a uninformed way -- it does not know what task you are reducing for. This is especially true for classification, where you have outright supervised information. But it also applies to clustering, where the space one would want to project to for clustering is better defined (for this algorithm) than just "have less dimensions). @usεr11852's answer talks about this. As I said dimensionality reduction does not know what task you are reducing for -- you inform it in your choice of which dimensionality reduction algorithm you to use.

So often rather than adding a dimensionality reduction step as preprocessing before clustering/classification, one is better to use a different classifier/cluster-er that incorperates a useful projection.

One thing dimentionality reduction does have going for it in this though is its unsupervised nature in creating the projection to the (hopefully) meaningful space. Which is useful if you have little label data. But there are often other methods that are closely linked to your classifier (e.g. for neural networks, using autoencoder e.g. deep belief network pretraining) that are going to work better, because they are designed with that final task in mind. Not the more general task of dimensionality reduction.