17
$\begingroup$

In as specific problem that I work with (a competition) I have the follwoing setting: 21 features (numerical on [0,1]) and a binary output. I have approx 100 K rows. The setting seems to be very noisy.

Me and other participants apply feature generation for a while and t-distributed stochastic neighbor embedding turned out to be rather powerful in this setting.

I stumbled upon this post "How to Use t-SNE Effectively" but still I can not really conclude on how to choose the hyperparameters best in my setting of classifcation.

Are there any rules of thumb (number of features, dimension of embedding -> choice of perplexity)?

I just apply ad-hoc settings at the moment as it takes too long to iterate various settings. Thanks for any comments.

$\endgroup$
1
  • $\begingroup$ This is a great question! Hopefully someone will find my answer lacklustre enough so you get another answer (and I learn something new too). $\endgroup$
    – usεr11852
    Nov 13, 2016 at 0:11

4 Answers 4

20
+50
$\begingroup$

I routinely use $t$-SNE (alongside clustering techniques - more on this in the end) to recognise/assess the presence of clusters in my data. Unfortunately to my knowledge there is no standard way to choose the correct perplexity aside looking at the produced reduced dimension dataset and then assessing if it is meaningful. There are some general facts, eg. distances between clusters are mostly meaningless, small perplexity values encourage small clot-like structures but that's about it.

A very rough rule-of-thumb is to check what is the error value associated with each reconstruction. $t$-SNE is trying to minimise the sum of the Kullback-Leibler divergences between the distribution of the distances between the data in the original domain and the distribution of distances between the data in the reduced dimension domain (actually the target distributions are the distributions of the probabilities that a point will pick another point as its neighbour but these are directly proportional to the distance between the two points). It could be argued that smaller values of KL-divergence show better results. This idea does not work very well in practice but it would theoretically help to exclude some ranges of the perplexity values as well as some runs of the algorithm that are clearly suboptimal. I explain why this heuristic is far from a panacea and how it could though be mildly useful: The perplexity parameter increases monotonically with the variance of the Gaussian used to calculate the distances/probabilities. Therefore as you increase the perplexity parameter as a whole you will get smaller distances in absolute terms and subsequent KL-divergence values. Nevertheless if you have 20 runs with the same perplexity and you cannot (do not want to) look at them you can always pick the one with the smallest variable hoping it retains the original distances more accurately. The same goes for the $\theta$, the approximation parameter for the Barnes-Hut approximation, assuming perplexity is fixed changing $\theta$ and then checking the resulting costs should be somewhat informative. In the end of the day, lower costs are associated with more faithful reconstructions. All is not lost though...

For your particular use case, a trick to mildly automate the procedure of picking a good perplexity value is the following: Run a small clustering procedure (say a $k$-means or DBSCAN) on the reduced dimensionality dataset and then assess the quality of that clustering using some sort of index (Cohen's $k$, Rand index, Fowlkes-Mallows, etc.) against what you try to predict. The idea here is that for your task at hand the correct representation of the data (the perplexity dependant $t$-SNE results) should give the most informative representation (in the form of one of the metrics mentioned) in terms of their alignment with the property you try to predict. This is why $t$-SNE was used in the first place after all, if the resulting representation is uninformative for the properties we are investigating then it is simply no good despite its low reconstruction error, visual appeal, etc. etc.

Let me point out that what I describe are heuristics. As mentioned in the beginning of my post, manually inspecting the results is an indispensable way of assessing the quality of the resulting dimensionality reduction/clustering.

$\endgroup$
1
  • $\begingroup$ Thank you for this. The idea of the index in how far the clustering fits the classification sounds interesting. $\endgroup$
    – Richi W
    Nov 13, 2016 at 16:29
5
$\begingroup$

We usually set the perplexity to 5% of the dataset size. So for a dataset with 100K rows I would start with perplexity of 5000, or at least 1000, if you don't have a high performance computer available. Our data sets are from flow cytometry analysis, they usually have 50k to 500k data points each with 10 to 20 numerical values.

$\endgroup$
4
$\begingroup$

It could be interesting for you to have a look the "Automatic Selection of t-SNE Perplexity" by Cao and Wang:

t-Distributed Stochastic Neighbor Embedding (t-SNE) is one of the most widely used dimensionality reduction methods for data visualization, but it has a perplexity hyperparameter that requires manual selection. In practice, proper tuning of t-SNE perplexity requires users to understand the inner working of the method as well as to have hands-on experience. We propose a model selection objective for t-SNE perplexity that requires negligible extra computation beyond that of the t-SNE itself. We empirically validate that the perplexity settings found by our approach are consistent with preferences elicited from human experts across a number of datasets. The similarities of our approach to Bayesian information criteria (BIC) and minimum description length (MDL) are also analyzed.

$\endgroup$
2
  • 2
    $\begingroup$ What were the conclusions..? $\endgroup$
    – Tim
    Dec 29, 2017 at 21:57
  • 1
    $\begingroup$ $S(Perplex.) = 2KL(P||Q) + \log(n)\frac{Perlex.}{n}$ (But +1 at Tim's comments,a paper's abstract is far from a complete answer; please try to built up an answer that is self-explanatory/contained.) $\endgroup$
    – usεr11852
    Dec 30, 2017 at 0:40
1
$\begingroup$

I found a very comprehensible article by Nikolay Oskolkov, a bioinfomatician and a medium-writer, explaining some really insightful heuristics on how to choose tSNE's hyperparameters.

How to tune hyperparameters of tSNE (by Nikolay Oskolkov, from Jul 19, 2019)

I hope you will find it useful too!

Just to put the summary of the article for your reference:

In this post we have learnt that despite tSNE can be sensitive with respect to its hyperparameters, there are simple rules for obtaining good looking tSNE plots for scRNAseq data. The optimal number of PCs for inputting into tSNE can be found through randomization of the expression matrix. The optimal perplexity can be calculated from the number of cells according to the simple power law Perplexity ~ N^(1/2). Finally, the optimal number of iterations should provide the largest distance between the data points of ~100 units.

However, this article only gives some rough heuristics for choosing a given set of hyperparameters values in a general setting. With regards to navigating through the hyperparameters' space while performing tSNE and choosing the best values for a particular dataset like your own, I agree with the aforementioned suggestion on using an intermediary procedure like k-means to judge the quality of clustering performed with respect to your concerned classification targets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.