# Why doesn't t-SNE need the labels before visualization?

In the original Maaten and Hinton paper, they explicitly say that the class membership is not used by the t-SNE calculations, only for picking colors in the plot.

For all of the data sets, there is information about the class of each datapoint, but the class information is only used to select a color and/or symbol for the map points. The class information is not used to determine the spatial coordinates of the map points.

The StatQuest video on t-SNE, however, makes it seem like the internal workings of the algorithm aims to keep the blue points (red points, orange points) in two dimensions near the blue points (red points, orange points) in one dimension, and I was using that to help me understand what the t-SNE algorithm does.

Is that because the blue points (red points, orange points) are all near one another in the higher dimension? If he used twelve different colors or four colors per cluster, then those clusters would still have to be clustered in the low dimension.

(That's got to be it, right? It sure would be nice to get confirmation after a major mind-blown moment, though.)

References

Maaten, Laurens van der, and Geoffrey Hinton. "Visualizing data using t-SNE." Journal of machine learning research 9.Nov (2008): 2579-2605.

t-SNE pairwise selects all the distances in the high-dimensional space and tries to preserve these in a lower-dimensional space. For a given point it calculates the distance from all other points in high-dimensional space, and gives each of these a probability (close points have a higher probability). It then does the same for all other points, and translates the closeness in the high-dimensional space onto closeness on a low-dimensional space. In the diagram, the red, orange and blue groups will have high probability within groups and lower between groups. I believe the “color” explanation given in the video is twofold:

1. To show the points before and after t-SNE.
2. To “point” to individual points more easily in the explanation. It is an iterative process to fix the points in the low-dimensional space so this may be the reason for needing to use the colors to point to individual points.
• So StatQuest could have used twelve different colors, showing that the colors near each other in $\mathbb{R}^2$ are near each other in $\mathbb{4}$ and that the distance colors in $\mathbb{R}^2$ are far from each other in $\mathbb{4}$.
– Dave
Aug 26, 2021 at 16:32
• Yes. All pairs are given a probability of how close they are in the high-dimensional domain and the low dimension points are positioned to match this pairwise probability distribution using KL-divergence. I think what you are thinking is you could have a dimension for class membership. There may be a problem with using class membership as a dimension input into t-SNE as it is a nominal variable. @amoeba helped my understanding with some implementation details and a linked paper stats.stackexchange.com/questions/495063/randomness-of-t-sne. Aug 26, 2021 at 18:59