10
$\begingroup$

I am very new to word embeddings. I want to visualize how the documents are looking after learning. I read that t-SNE is the approach to do it. I have 100K documents with 250 dimensions as size of the embedding. There are several packages available as well.

However, for t-SNE, I don't know how many iterations or the value of alpha or the value of perpexility I should keep to learn better.

Are these hyper-parameters or can these be determined by some attributes?

$\endgroup$
12
$\begingroup$

I highly reccomend the article How to Use t-SNE Effectively. It has great animated plots of the tsne fitting process, and was the first source that actually gave me an intuitive understanding of what tsne does.

At a high level, perplexity is the parameter that matters. It's a good idea to try perplexity of 5, 30, and 50, and look at the results.

But seriously, read How to Use t-SNE Effectively. It will make your use of TSNE more effective.

For packages, use Rtsne in R, or sklearn.manifold.TSNE in python

$\endgroup$
2
$\begingroup$

I will cite the FAQ from t-SNE website. First for perplexity:

How should I set the perplexity in t-SNE?

The performance of t-SNE is fairly robust under different settings of the perplexity. The most appropriate value depends on the density of your data. Loosely speaking, one could say that a larger / denser dataset requires a larger perplexity. Typical values for the perplexity range between 5 and 50.

For all other paremeters I would consider reading this:

How can I asses the quality of the visualizations that t-SNE constructed?

Preferably, just look at them! Notice that t-SNE does not retain distances but probabilities, so measuring some error between the Euclidean distances in high-D and low-D is useless. However, if you use the same data and perplexity, you can compare the Kullback-Leibler divergences that t-SNE reports. It is perfectly fine to run t-SNE ten times, and select the solution with the lowest KL divergence.

In other words it means: look at the plot, if the visualization is good don't change the parameters. You can also choose the run with the lowest KL divergence for each fixed perplexity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.