41
$\begingroup$

I have a PhD in molecular biology. My studies recently started to involve high dimensional data analysis. I got the idea of how t-SNE works (thanks to a StatQuest video on YouTube) but can't seem to wrap my mind around UMAP (I listened to the UMAP creator's talk online but didn't find it easy to understand). I went back to original paper describing it but it was too much math for me.

Can anybody shed some light on the issue? I am looking or an intuitive explanation, similar to the StatQuest video linked above.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I'm looking for intuition in words but also some simple insight into mathematical calculations (I don't know if the latter is possible). I'd like to see something like this for UMAP: "StatQuest tSNE Clearly Explained" youtube.com/watch?v=NEaUSP4YerM When I say, I understand how tSNE works, I'm referring to the broad calculation approach described in the video. It is a bit difficult for me to imagine the example in the video in a higher dimensional space, but overall I can see how the distances are calculated. I'd like to have a similar understanding about UMAP $\endgroup$
    – Atakan
    Commented Apr 12, 2019 at 19:02
  • 1
    $\begingroup$ worth reading > towardsdatascience.com/how-exactly-umap-works-13e3040e1668 $\endgroup$
    – jjrr
    Commented Apr 16, 2020 at 18:13
  • 1
    $\begingroup$ FYI, a recent StatQuest video on UMAP has a short section at the end, comparing it to t-SNE: youtube.com/watch?v=eN0wFzBA4Sc $\endgroup$
    – karpfen
    Commented Mar 24, 2022 at 14:15

2 Answers 2

Reset to default
34
$\begingroup$

You said that your understanding of t-SNE is based on https://www.youtube.com/watch?v=NEaUSP4YerM and you are looking for an explanation of UMAP on a similar level.

I watched this video and it is pretty accurate in what it says (I have some minor nitpicks, but overall it is fine). Funny enough, it almost applies to UMAP just as it is. Here are things that do not apply:

  1. Similarities are computed from distances using a different kernel; it is not Gaussian, but it also decays exponentially and it also has adaptive width, as in t-SNE.
  2. Similarities are not normalized to sum to 1, but still end up being normalized to sum a constant value.
  3. Similarities are symmetrized, but not just by averaging.
  4. The similarity kernel in the embedding space is not exactly t-distribution kernel, but a very very similar kernel.

I think all of these differences are not very important and not very consequential. The actually important part is the part where in the video the narrator says (10m40s):

We want to make this row look like this row [...]

The video does not explain how t-SNE quantifies whether they are similar or not and how it goes on achieving that they look similar. Both parts are different in UMAP. But the quoted statement can apply to UMAP too.

The way the UMAP paper is written, the computational similarities to t-SNE are not very apparent. Scroll down to Appendix C in https://arxiv.org/pdf/1802.03426.pdf and/or look here https://jlmelville.github.io/uwot/umap-for-tsne.html, (Link to post itself) if you want to see a side-by-side comparison of the computations that I list above and the loss functions of t-SNE and UMAP.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is very helpful, thanks! I have a question about that particular segment of the video. When he is showing the "unordered heatmap" on the left, the annotation dots (colorful data points) are in order and the color intensity at the row-column intersection don't match to the graph on the right side. That is a misrepresentation, right? I'd expect the graph on the left to be unordered when it comes to data points, which will then be ordered by UMAP. Am I on the wrong track here? $\endgroup$
    – Atakan
    Commented Apr 13, 2019 at 6:25
  • $\begingroup$ @Atakan I am not quite sure what you are saying. I don't see a misrepresentation. I'm looking at the video frame at 10:40. The left similarity matrix is "a mess". The "annotation dots" to the left simply mark the cluster of each point; imagine that the points are numbered from 1 to 12. The 12 rows/columns of the matrix correspond to these points; the first 4 rows correspond to the "blue" points, the next 4 correspond to the "red" points, etc. Since the 1-dimensional embedding (in the bottom of the frame) is "a mess", the similarities in the matrix are also "a mess". $\endgroup$
    – amoeba
    Commented Apr 13, 2019 at 20:33
29
$\begingroup$

The main difference between t-SNE and UMAP is the interpretation of the distance between objects or "clusters". I use the quotation marks since both algorithms are not meant for clustering - they are meant for visualization mostly.

t-SNE preserves local structure in the data.

UMAP claims to preserve both local and most of the global structure in the data.

This means with t-SNE you cannot interpret the distance between clusters A and B at different ends of your plot. You cannot infer that these clusters are more dissimilar than A and C, where C is closer to A in the plot. But within cluster A, you can say that points close to each other are more similar objects than points at different ends of the cluster image.

With UMAP, you should be able to interpret both the distances between / positions of points and clusters.

Both algorithms are highly stochastic and very much dependent on choice of hyperparameters (t-SNE even more than UMAP) and can yield very different results in different runs, so your plot might obfuscate an information in the data that a subsequent run might reveal.

Good old PCA on the other hand is deterministic and easily understandable with basic knowledge of linear algebra (matrix multiplication and eigenproblems), but is just a linear reduction in contrast to the non-linear reductions of t-SNE and UMAP.

Improve this answer
$\endgroup$
5
  • 13
    $\begingroup$ I strongly disagree with this assessment: "t-SNE preserves local structure and ignores global structure. UMAP acknowledges both local and global structure." UMAP operates on the k-nearest neighbours graph (for some small value of k), exactly as t-SNE does. $\endgroup$
    – amoeba
    Commented Apr 12, 2019 at 15:37
  • 2
    $\begingroup$ This is actually what the authors of the UMAP claim, see e.g. here or here. Do you know of a comparison (theoretical or practical) that shows that their claim isn't true? Please share! $\endgroup$
    – Edgar
    Commented Apr 12, 2019 at 15:50
  • 8
    $\begingroup$ I know that they say this... :-/ But it's them who are making this statement, so the onus is on them to prove that (not on me to disprove). I wasn't convinced by what I saw so far. $\endgroup$
    – amoeba
    Commented Apr 12, 2019 at 15:54
  • 2
    $\begingroup$ true, it's still kind of a new method. let's hope some more rigorous assessment of umap vs t-sne will be done. i changed my answer to reflect your point. $\endgroup$
    – Edgar
    Commented Apr 12, 2019 at 16:00
  • 13
    $\begingroup$ There is now a preprint on this very topic: UMAP does not preserve global structure any better than t-SNE when using the same initialization $\endgroup$
    – krassowski
    Commented Apr 18, 2020 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.