HDBSCAN: most data clustered as noise (-1) I am trying to perform topic modeling on text data, ie. cluster the text messages by topic. I am approaching this by using a BERT model to get sentence embeddings, then use T-sne to reduce the dimensionality of the embeddings to an 8-dimensional space. In the end, I use HDBSCAN to cluster the dimensionally-reduced embeddings.
When i do so, about 40% of the data points in the train set are labelled/clustered as -1 (noise). When predicting on new data, 60% of points get labelled as -1. This is really high fraction because i know most of the data should belong to a topic, and I am also setting the HDBSCAN parameter min_samples = 1.
I have seen other people also face such issue with hdbscan. Does this scenario mean that i might be doing something wrong and need to make more adjustments to my data, or such a scenario is typical for hdbscan?
If the latter is true, should i resort to soft-clustering with hdbscan, or try another clustering method completely?
 A: there are two parameters to set in dbscan, one minPts, another is distance eps for searching neighbors.
https://en.wikipedia.org/wiki/DBSCAN
Make the eps larger and make the minPts smaller, will solve your problem (most data are not in clusters). What you did is setting the minPts to 1, but not setting the eps.
Try a larger number on eps
A: HDBSCAN is a density-based clustering algorithm. By using t-SNE, you are inherently losing out on information, since the algorithm preserves neither distances nor density, both of which are essential for HDBSCAN. Since you are interested in preserving the density for HDBSCAN,
Since you need to reduce the dimensionality of your embeddings, you may want to look into using DensMAP or UMAP in place of t-SNE. see link.
Additionally, it could also be of value to try higher values for your min_samples parameter. HDBSCAN will use the min_samples parameter to estimate a probability density function for your data. Using a very low value for min_samples will result in a very noisy estimate for the PDF while using a very high value will result in the finer details of the PDF being lost.
If you'd like to look into it more, consider taking a look at this article that offers some insight into the workings of HDBSCAN.
