t-SNE method does not require the removal of duplicates. The fact that it is a default feature in
Rtsne does not imply its requirement.
It is useful for some short-term event monitoring.
For characterising long-term trends and/or patterns with big data sets, I see little use.
Rtsne default setup can be more inclined for characterising events in the time-domain, without any studies in Fourier domain.
Assume you have points in the time-domain. The duplicate algorithm causes significant amount of false positives because the duplicate checking is mostly designed on the time-domain signal. Fourier space can show that those events which are considered by the algorithm duplicate are not necessary so.
So my observation is that the algorithm is greedy about duplicate points in the time-domain, which is not useful for me when considering long-term signals, long-terms trends and long-term patterns.
The fact that the point is duplicate in the time-domain does not actually mean that it is duplicate also in Fourier domain.
I think it will be more a coincidence if is a duplicate in a time domain in the real-life applications.
So turning off the feature, should be ok.
To estimate how much of the points are really duplicates in both domains is specific on the case study.
I get significantly better descriptors of events and/or phenomena by considering long-term data sets without the duplicate check in many real-life applications.
I think the
Rtsne documentation is not clear about the case in saying [turn off
check_duplicates and] don't wast processing power.
There are really other reasons as described above why the
check_duplicates can be turned off as realised also by some other implementations of the method.
check_duplicates=TRUE is a personal selection of the
Rtsne developer by default at the moment.
I would love to hear if there is any implementation reasons for the decision.