Why does t-SNE not separate linearly separable classes?

Question

I have small convolution neural net for classification EEG patterns (2 class problem) with several conv layers and one dense layer. This net performs pretty well. For education purposes I used convolutional part of this pretrained net (all net except dense layer) to extract features from training samples. So this new data representation lineary seperable (because dense layer can correctly classify them) But when I used TSNE on this training set representation and got only one blob and points from different classes where mixed.

This situation repeats independently from TSNE parameters. Does it possible to get only one cluster with mixed classes for linearly separable dataset?

Answer 1

Yes.

You can use the following code to convince yourself.

N <- 1000
P <- 3

# Generates some random data
data <- matrix(data = rnorm(N*P), nrow = N, ncol = P)

# Assignate linearly separable classes
labels <- (data[,1]+data[,2]>0)+1.

# Make sure that the data can be separated
plot(data[,1],data[,2], col = labels, xlab = 'x1', ylab = 'x2')

require(Rtsne)
model <- Rtsne::Rtsne(data)

# Observe this result while varying P
plot(model$Y, col = labels, type = 'p', pch = 21, xlab = 'tSNE x_1', ylab = 'tSNE x_2')

This is what you would observe when P is 3 (only one irrelevant attribute with respect to the linear separation, we are close to reproducing the linear separation).

And for P is 15, the tSNE cannot reproduce the linear separation.

What happened ?

This is simple. The tSNE method relies on pairwise distances between points to produce clusters and is therefore totally unaware of any possible linear separability of your data.

If your points are "close" to each other, on different sides of a "border", a tSNE will consider that they belong to a same cluster.

This was exactly the point of the simulations above. When the number of dimensions is large, points look close to each other, no matter the side of the border they belong to. This is what tSNE fails to capture here.

On the other hand, when the number of irrelevant dimension is low, close points had "no other choice" but to be on the same side of the border.

Side note.

Even though you may have a nice performance with a neural network, it may not mean that your data is linearly separable (unless there is only one unit in your neural, hum, network). Indeed neural networks can recognize non linear boundaries. If you want to test how "linearly separable" a data set is, you should use linear Support Vector Machines or regressions.