Why doesn't t-SNE need the labels before visualization?

In the original Maaten and Hinton paper, they explicitly say that the class membership is not used by the t-SNE calculations, only for picking colors in the plot.

For all of the data sets, there is information about the class of each datapoint, but the class information is only used to select a color and/or symbol for the map points. The class information is not used to determine the spatial coordinates of the map points.

The StatQuest video on t-SNE, however, makes it seem like the internal workings of the algorithm aims to keep the blue points (red points, orange points) in two dimensions near the blue points (red points, orange points) in one dimension, and I was using that to help me understand what the t-SNE algorithm does.

Is that because the blue points (red points, orange points) are all near one another in the higher dimension? If he used twelve different colors or four colors per cluster, then those clusters would still have to be clustered in the low dimension.

(That's got to be it, right? It sure would be nice to get confirmation after a major mind-blown moment, though.)

References

Maaten, Laurens van der, and Geoffrey Hinton. "Visualizing data using t-SNE." Journal of machine learning research 9.Nov (2008): 2579-2605.

• So StatQuest could have used twelve different colors, showing that the colors near each other in $\mathbb{R}^2$ are near each other in $\mathbb{4}$ and that the distance colors in $\mathbb{R}^2$ are far from each other in $\mathbb{4}$.