A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Hot answers tagged

108

The problem with t-SNE is that it does not preserve distances nor density. It only to some extent preserves nearest-neighbors. The difference is subtle, but affects any density- or distance based algorithm. To see this effect, simply generate a multivariate Gaussian distribution. If you visualize this, you will have a ball that is dense and gets much less ...


85

KL divergence is a natural way to measure the difference between two probability distributions. The entropy $H(p)$ of a distribution $p$ gives the minimum possible number of bits per message that would be needed (on average) to losslessly encode events drawn from $p$. Achieving this bound would require using an optimal code designed for $p$, which assigns ...


68

$t$-SNE is a great piece of Machine Learning but one can find many reasons to use PCA instead of it. Of the top of my head, I will mention five. As most other computational methodologies in use, $t$-SNE is no silver bullet and there are quite a few reasons that make it a suboptimal choice in some cases. Let me mention some points in brief: Stochasticity of ...


35

I would like to provide a somewhat dissenting opinion to the well argued (+1) and highly upvoted answer by @ErichSchubert. Erich does not recommend clustering on the t-SNE output, and shows some toy examples where it can be misleading. His suggestion is to apply clustering to the original data instead. use t-SNE for visualization (and try different ...


34

The main reason that $t$-SNE is not used in classification models is that it does not learn a function from the original space to the new (lower) dimensional one. As such, when we would try to use our classifier on new / unseen data we will not be able to map / pre-process these new data according to the previous $t$-SNE results. There is work on training ...


20

Individual axes in t-SNE have no meaning at all. Algorithms such as MDS, SNE, t-SNE, etc. only care about pairwise distances between points. They try to position the points on a plane such that the pairwise distances between them would minimize a certain criterion. This means that if you take a t-SNE plot and rotate it, then the resulting arrangement will ...


19

PCA selects influential dimensions by eigenanalysis of the N data points themselves, while MDS selects influential dimensions by eigenanalysis of the $N^2$ data points of a pairwise distance matrix. This has the effect of highlighting the deviations from uniformity in the distribution. Considering the distance matrix as analogous to a stress tensor, MDS may ...


18

Centering shouldn't matter since the algorithm only operates on distances between points, however rescaling is necessary if you want the different dimensions to be treated with equal importance, since the 2-norm will be more heavily influenced by dimensions with large variance.


17

I routinely use $t$-SNE (alongside clustering techniques - more on this in the end) to recognise/assess the presence of clusters in my data. Unfortunately to my knowledge there is no standard way to choose the correct perplexity aside looking at the produced reduced dimension dataset and then assessing if it is meaningful. There are some general facts, eg. ...


16

Definitely not. I agree that t-SNE is an amazing algorithm that works extremely well and that was a real breakthrough at the time. However: it does have serious shortcomings; some of the shortcomings must be solvable; there already are algorithms that perform noticeably better in some cases; many t-SNE's properties are still poorly understood. Somebody ...


15

I had exactly the same question and posted it on a YouTube video of a CS231n lecture given by Andrej Karpathy a few weeks ago. Here is the question I posted followed by Andrej' response: https://www.youtube.com/watch?v=ta5fdaqDT3M&lc=z12ji3arguzwgxdm422gxnf54xaluzhcx Q: Does t-SNE need an entire batch of images (or more generally, data) to create ...


13

T-Sne is a reduction technique that maintains the small scale structure (i.e. what is particularly close to what) of the space, which makes it very good at visualizing data separability. This means that T-Sne is particularly useful for early visualization geared at understanding the degree of data separability. Other techniques (PCA for example) leave data ...


13

https://stats.stackexchange.com/a/249520/7828 is an excellent general answer. I'd like to focus a bit more on your problem. You apparently want to see how your samples relate with respect to your 7 input variables. That is something t-SNE doesn't do. The idea of SNE and t-SNE is to place neighbors close to each other, (almost) completly ignoring the global ...


13

Great question. I will answer it using t-SNE because I assume it is familiar to more people. I think UMAP is very promising and is a great contribution but to be honest I am getting a little bit annoyed with all the marketing and the hype that surrounds it. People think that t-SNE cannot embed new points but UMAP miraculously can. In reality, t-SNE can do it ...


12

I highly reccomend the article How to Use t-SNE Effectively. It has great animated plots of the tsne fitting process, and was the first source that actually gave me an intuitive understanding of what tsne does. At a high level, perplexity is the parameter that matters. It's a good idea to try perplexity of 5, 30, and 50, and look at the results. But ...


11

When dealing with streaming data, you might not want/need to embed all the points in history in a single t-SNE map. As an alternative, you can perform an online embedding by following these simple steps: choose a time-window of duration T, long enough so that each pattern of interest appears at least a couple of times in the window duration. scroll the ...


11

You said that your understanding of t-SNE is based on https://www.youtube.com/watch?v=NEaUSP4YerM and you are looking for an explanation of UMAP on a similar level. I watched this video and it is pretty accurate in what it says (I have some minor nitpicks, but overall it is fine). Funny enough, it almost applies to UMAP just as it is. Here are things that ...


10

Out of the box, tSNE has a few hyperparameters, the main one being perplexity. Remember that heuristically, perplexity defines a notion of similarity for tSNE and a universal perplexity is used for all data-points. You could try generating a labelled dataset where each cluster has wildly different perplexity. This can be accomplished by doing a mixture of ...


10

You have to understand what TSNE does before you use it. It starts by building a neighboorhood graph between feature vectors based on distance. The graph connects a node(feature vector) to its n nearest nodes(in terms of distance in feature space). This n is called the perplexity parameter. The purpose of building this graph is rooted in the sort of ...


9

It is hard to compare these approaches. PCA is parameter free. Given the data, you just have to look at the principal components. On the other hand, t-SNE relies on severe parameters : perplexity, early exaggeration, learning rate, number of iterations - though default values usually provide good results. So you can't just compare them, you have to ...


9

Three general remarks: t-SNE is excellent at preserving cluster structure but is not very good at preserving continuous "manifold structure". One famous toy example is the Swiss roll data set, and it is well-known that t-SNE has trouble "unrolling" it. In fact, one can use t-SNE to unroll it, but one has to be really careful with choosing optimisation ...


8

Consider the following dataset: PC1 axis is maximizing the variance of the projection. So in this case it will obviously go diagonally from lower-left to upper-right corner: The largest pairwise distance in the original dataset is between these two outlying points; notice that it is almost exactly preserved in the PC1. Smaller but still substantial ...


8

Question 1: Let's say you have observed a data matrix $X \in \mathbb R^{n \times p}$. From this you can compute the eigendecomposition $X^T X = Q \Lambda Q^T$. The question now is: if we get new data coming from the same population, perhaps collected into a matrix $Z \in \mathbb R^{m \times p}$, will $ZQ$ be close to the ideal orthogonal rotation of $Z$? ...


7

Both of them try to find a lower dimensionality embedding of your data. However, there are different minimization problems. More specifically, an autoencoder tries to minimize the reconstruction error, while t-SNE tries to find a lower dimensional space and at the same time it tries to preserve the neighborhood distances. As a result of this attribute, t-SNE ...


7

There is a recently published variant, called A-tSNE, which supports dynamically adding new data and refining clusters either based on interest areas or by user input. The paper linked below has some pretty nice examples of this: Citation: arXiv:1512.01655 Approximated and User Steerable tSNE for Progressive Visual Analytics Nicola Pezzotti, Boudewijn ...


7

Unfortunately, no; comparing the optimality of a perplexity parameter through the correspond $KL(P||Q)$ divergence is not a valid approach. As I explained in this question: "The perplexity parameter increases monotonically with the variance of the Gaussian used to calculate the distances/probabilities $P$. Therefore as you increase the perplexity parameter ...


7

I think with large perplexity t-SNE can reconstruct the global topology, as indicated in https://distill.pub/2016/misread-tsne/. From the fish image, I sampled 4000 points for t-SNE. With a large perplexity (2000), the fish image was virtually reconstructed. Here is the original image. Here is the image reconstructed by t-SNE with perplexity = 2000.


7

Here's an excellent analysis of how varying the parameters when running t-SNE affects some very simple datasets: http://distill.pub/2016/misread-tsne/. In general, t-SNE seems to do well at recognizing high-dimensional structures (including relationships more complex than clusters), though this is subject to parameter tuning, especially perplexity values.


7

I would still love to hear other comments but I'll post my own answer for now, as I see it. While I was looking for a more "practical" answer, there are two theoretical "dis-advantages" to t-sne which are worth mentioning; the first one is less problematic, and the second should definitely be considered: t-sne cost function is not convex, so we are not ...


7

First a brief answer, and then a longer comment: Answer SNE techniques compute an N ×N similarity matrix in both the original data space and in the low-dimensional embedding space in such a way that the similarities form a probability distribution over pairs of objects. Specifically, the probabilities are generally given by a normalized Gaussian kernel ...


Only top voted, non community-wiki answers of a minimum length are eligible