I want to see how 7 measures of text correction behaviour (time spent correcting the text, number of keystrokes, etc.) relate to each other. The measures are correlated. I ran a PCA to see how the measures projected onto PC1 and PC2, which avoided the overlap of running separate two-way correlation tests between the measures.

I was asked why not using t-SNE, since the relationship between some of the measures might be non-linear.

I can see how allowing for non-linearity would improve this, but I wonder if there is any good reason to use PCA in this case and not t-SNE? I'm not interested in clustering the texts according to their relationship to the measures, but rather in the relationship between the measures themselves.

(I guess EFA could also a better/another approach, but that's a different discussion.) Compared to other methods, there are few posts on here about t-SNE, so the question seems worth asking.

  • 3
    t-SNE does not scale well with size of dataset, while PCA does. This comes from experience in running both of them on large dataset using scikit-learn implementation. – Mai Oct 5 '16 at 17:15
  • @Mai presumably this applies mostly to large datasets? My dataset is on the small side (a few hundred data points). – user3744206 Oct 7 '16 at 10:27

$t$-SNE is a great piece of Machine Learning but one can find many reasons to use PCA instead of it. Of the top of my head, I will mention five. As most other computational methodologies in use, $t$-SNE is no silver bullet and there are quite a few reasons that make it a suboptimal choice in some cases. Let me mention some points in brief:

  1. Stochasticity of final solution. PCA is deterministic; $t$-SNE is not. One gets a nice visualisation and then her colleague gets another visualisation and then they get artistic which looks better and if a difference of $0.03\%$ in the $KL(P||Q)$ divergence is meaningful... In PCA the correct answer to the question posed is guaranteed. $t$-SNE might have multiple minima that might lead to different solutions. This necessitates multiple runs as well as raises questions about the reproducibility of the results.

  2. Interpretability of mapping. This relates to the above point but let's assume that a team has agreed in a particular random seed/run. Now the question becomes what this shows... $t$-SNE tries to map only local / neighbours correctly so our insights from that embedding should be very cautious; global trends are not accurately represented (and that can be potentially a great thing for visualisation). On the other hand, PCA is just a diagonal rotation of our initial covariance matrix and the eigenvectors represent a new axial system in the space spanned by our original data. We can directly explain what a particular PCA does.

  3. Application to new/unseen data. $t$-SNE is not learning a function from the original space to the new (lower) dimensional one and that's a problem. On that matter, $t$-SNE is a non-parametric learning algorithm so approximating with parametric algorithm is an ill-posed problem. The embedding is learned by directly moving the data across the low dimensional space. That means one does not get an eigenvector or a similar construct to use in new data. In contrast, using PCA the eigenvectors offer a new axes system what can be directly used to project new data. [Apparently one could try training a deep-network to learn the $t$-SNE mapping (you can hear Dr. van der Maaten at ~46' of this video suggesting something along this lines) but clearly no easy solution exists.]

  4. Incomplete data. Natively $t$-SNE does not deal with incomplete data. In fairness, PCA does not deal with them either but numerous extensions of PCA for incomplete data (eg. probabilistic PCA) are out there and are almost standard modelling routines. $t$-SNE currently cannot handle incomplete data (aside obviously training a probabilistic PCA first and passing the PC scores to $t$-SNE as inputs).

  5. The $k$ is not (too) small case. $t$-SNE solves a problem known as the crowding problem, effectively that somewhat similar points in higher dimension collapsing on top of each other in lower dimensions (more here). Now as you increase the dimensions used the crowding problem gets less severe ie. the problem you are trying to solve through the use of $t$-SNE gets attenuated. You can work around this issue but it is not trivial. Therefore if you need a $k$ dimensional vector as the reduced set and $k$ is not quite small the optimality of the produce solution is in question. PCA on the other hand offer always the $k$ best linear combination in terms of variance explained. (Thanks to @amoeba for noticing I made a mess when first trying to outline this point.)

I do not mention issues about computational requirements (eg. speed or memory size) nor issues about selecting relevant hyperparameters (eg. perplexity). I think these are internal issues of the $t$-SNE methodology and are irrelevant when comparing it to another algorithm.

To summarise, $t$-SNE is great but as all algorithms has its limitations when it comes to its applicability. I use $t$-SNE almost on any new dataset I get my hands on as an explanatory data analysis tool. I think though it has certain limitations that do not make it nearly as applicable as PCA. Let me stress that PCA is not perfect either; for example, the PCA-based visualisations are often inferior to those of $t$-SNE.

  • @amoeba: I removed the point because it was getting too laborious; I was mostly motivated by the idea of $t$-SNE having issues with the crowding problem being less severe when using higher dimensions (instead of $k={2,3,4}$) and thus offering muddled insights but I mixed up the point I was trying to make. Also, as reconstruction is possible from LLE (Roweis & Saul, 2000) why wouldn't it be possible by t-SNE? – usεr11852 Dec 4 '16 at 1:37
  • @amoeba: Thank you for mentioning it. I updated my answer accordingly. – usεr11852 Dec 4 '16 at 1:56
  • 2
    Regarding your point #3: here is the 2009 paper on parametric t-sne lvdmaaten.github.io/publications/papers/AISTATS_2009.pdf. It seems it did not really take off (it has 25 times less citations than the original t-sne paper), but in fact it's quite easy to implement with today's technology/libraries. I have it up and running in Keras; I've been working on investigating (and possibly extending) it in the last weeks. – amoeba Apr 27 '17 at 9:01
  • Cool! (+1) If you get an arXiv pre-print floating please let me know (here or 10-fold), I will be very curious about the results. Yes, I have seen that paper at the time of writing this answer (it is actually a well-known paper I would say) but as you said it did not seem to be taken up. Also point #3 remains perfectly valid: you need to build a DNN to get out something PCA offers through a single matrix crossproduct. – usεr11852 Apr 27 '17 at 18:52

https://stats.stackexchange.com/a/249520/7828

is an excellent general answer.

I'd like to focus a bit more on your problem. You apparently want to see how your samples relate with respect to your 7 input variables. That is something t-SNE doesn't do. The idea of SNE and t-SNE is to place neighbors close to each other, (almost) completly ignoring the global structure.

This is excellent for visualization, because similar items can be plotted next to each other (and not on top of each other, c.f. crowding).

This is not good for further analysis. Global structure is lost, some objects may have been blocked from moving to their neighbors, and separation between different groups is not preserved quantitatively. Which is largely why e.g. clustering on the projection usually does not work very well.

PCA is quite the opposite. It tries to preserve the global properties (eigenvectors with high variance) while it may lose low-variance deviations between neighbors.

  • Ah that's exactly what I assumed. I'm not interested in how the data points are located in the space, but rather on how the measures themselves are related to each other. These two things are connected, of course, but in terms of visualising and interpreting these relationships, I suspect only PCA does what I want. For example, there are both positive and negative relationships between the measures and what I'm really interested in is the absolute value of the associations, which again I think is easier to interpret/see if I use PCA. – user3744206 Dec 9 '16 at 9:11
  • For that use case it may be better to rather look at the correlation matrix itself, i.e. only do pairwise comparisons. Then you can also handle nonlinearity, e.g. by using spearman correlation. – Anony-Mousse Dec 10 '16 at 9:52

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.