Document clustering with DBSCAN

Question

I'm building a document clustering tool which will be fed with datasets of variable size (from several hundred to several million). Documents are represented as dense vectors in (around) 100-dimensional space with a custom metric. They mostly come from a restricted, quite narrow domain, so the "real" clusters may be overlapping, and probably not too obvious. There may be a considerable amount of noise, too. Number of clusters doesn't have to be fixed, but I'd like to have some control over it.

I started out with K-Means, the results were not bad, but I hoped it could be improved. Generally I'm aiming for high precision and I'm perfectly okay with some of the noise left unclustered, so I thought about DBSCAN. However, using fixed DBSCAN parameters doesn't seem to be a good idea, because, if I understand correctly, larger dataset (as I said the size is not known in advance) will mean higher density → and probably broken clustering at some point (is this correct?). Therefore I have following questions:

1. Is it possible and reasonable to try to estimate DBSCAN parameters on the fly based on data size/density? If so, how can I approach this?

2. Is it a good idea at all to use DBSCAN in this setting? Would you recommend something more suitable?

3. Are there any general tips I should keep in mind for improving clustering precision? (That is, filtering out noise → getting separate "noise clusters" or "spam clusters" is also fine.)

Footnote: I'm well aware of topic modelling algorithms like LDA or NMF, but so far none of them worked well for me.

Answer 1

Choosing DBSCAN parameters that would suit all of your data sets will likely not work. Plus, you said that clusters could overlap.

Maybe you should research all those follow-up algorithms (DBSCAN is 20 years old) such as OPTICS and HDBSCAN* if they better suit your problem. With 100 dimensions, subspace approaches (which often allow overlapping clusters) are worth looking at, too.