Why does the implementation of t-SNE in R default to the removal of duplicates?

Question

Specifically, the R implementation of t-SNE in the Rtsne package has a "check_duplicates" argument, and the documentation suggests that "it is best to make sure there are no duplicates present and set this option to FALSE, especially for large datasets".

Further, if you attempt to run t-SNE on a dataset in R that does have duplicates, you get the error message: "Error in [command snipped by user]: Remove duplicates before running TSNE."

So, why does this behavior occur? I have a dataset in which multiple samples coincidentally have the same measurements.

Is it simply a "duplicate datapoints, after reduction, will have the same data points anyway, so don't waste processing power"? Does the presence of duplicates affect the process' calculations?

Answer 1

t-SNE method does not require the removal of duplicates. The fact that it is a default feature in Rtsne does not imply its requirement. It is useful for some short-term event monitoring. For characterising long-term trends and/or patterns with big data sets, I see little use. The Rtsne default setup can be more inclined for characterising events in the time-domain, without any studies in Fourier domain.

Assume you have points in the time-domain. The duplicate algorithm causes significant amount of false positives because the duplicate checking is mostly designed on the time-domain signal. Fourier space can show that those events which are considered by the algorithm duplicate are not necessary so.

So my observation is that the algorithm is greedy about duplicate points in the time-domain, which is not useful for me when considering long-term signals, long-terms trends and long-term patterns. The fact that the point is duplicate in the time-domain does not actually mean that it is duplicate also in Fourier domain. I think it will be more a coincidence if is a duplicate in a time domain in the real-life applications. So turning off the feature, should be ok. To estimate how much of the points are really duplicates in both domains is specific on the case study. I get significantly better descriptors of events and/or phenomena by considering long-term data sets without the duplicate check in many real-life applications.

I think the Rtsne documentation is not clear about the case in saying [turn off check_duplicates and] don't wast processing power. There are really other reasons as described above why the check_duplicates can be turned off as realised also by some other implementations of the method. The check_duplicates=TRUE is a personal selection of the Rtsne developer by default at the moment. I would love to hear if there is any implementation reasons for the decision.

Answer 2

The algorithm is designed to handle datasets without duplicate information, so the package mades a check before apply the technique. They suggest you to remove duplicates and set check_duplicates = FALSE for a performance improvement.

The implementation in R is this:

if (check_duplicates & !is_distance){
if (any(duplicated(X))) { stop("Remove duplicates before running TSNE.")}

Whith default values check_duplicates = TRUE and is_distance = FALSE.

The paper, for who wants to undestand more about the method, is here.

Answer 3

In addition to Léo's answer it can even be 'dangerous' to remove duplicates.

Imagine you want to explore a dataset describing a problem for machine learning to see if there is any correlation between the features and the labels:

If you remove the duplicates in the feature space, tsne has a higher chance to create clusters with specific labels. But if there is no correlation between features and labels (i.e. the same feature-combination can result in different labels) this may tric you into thinking there is a correlation because these duplicate points were removed and your tsne-plot will look 'cleaner'.