1
$\begingroup$

I'm building a document clustering tool which will be fed with datasets of variable size (from several hundred to several million). Documents are represented as dense vectors in (around) 100-dimensional space with a custom metric. They mostly come from a restricted, quite narrow domain, so the "real" clusters may be overlapping, and probably not too obvious. There may be a considerable amount of noise, too. Number of clusters doesn't have to be fixed, but I'd like to have some control over it.

I started out with K-Means, the results were not bad, but I hoped it could be improved. Generally I'm aiming for high precision and I'm perfectly okay with some of the noise left unclustered, so I thought about DBSCAN. However, using fixed DBSCAN parameters doesn't seem to be a good idea, because, if I understand correctly, larger dataset (as I said the size is not known in advance) will mean higher density → and probably broken clustering at some point (is this correct?). Therefore I have following questions:

  1. Is it possible and reasonable to try to estimate DBSCAN parameters on the fly based on data size/density? If so, how can I approach this?

  2. Is it a good idea at all to use DBSCAN in this setting? Would you recommend something more suitable?

  3. Are there any general tips I should keep in mind for improving clustering precision? (That is, filtering out noise → getting separate "noise clusters" or "spam clusters" is also fine.)

Footnote: I'm well aware of topic modelling algorithms like LDA or NMF, but so far none of them worked well for me.

$\endgroup$
1
$\begingroup$

Choosing DBSCAN parameters that would suit all of your data sets will likely not work. Plus, you said that clusters could overlap.

Maybe you should research all those follow-up algorithms (DBSCAN is 20 years old) such as OPTICS and HDBSCAN* if they better suit your problem. With 100 dimensions, subspace approaches (which often allow overlapping clusters) are worth looking at, too.

$\endgroup$
  • $\begingroup$ Thanks for the answer! Could you elaborate a bit on the last part (subspace approaches)? What's the idea behind it? Any specific algorithms I could look for? $\endgroup$ – machaerus Mar 16 '17 at 23:10
  • $\begingroup$ There are surveys on this, if I am not mistaken. SUBCLU and PreDeCon are example DBSCAN variants. $\endgroup$ – Anony-Mousse Mar 17 '17 at 6:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.