I'm building a document clustering tool which will be fed with datasets of variable size (from several hundred to several million). Documents are represented as dense vectors in (around) 100-dimensional space with a custom metric. They mostly come from a restricted, quite narrow domain, so the "real" clusters may be overlapping, and probably not too obvious. There may be a considerable amount of noise, too. Number of clusters doesn't have to be fixed, but I'd like to have some control over it.
I started out with K-Means, the results were not bad, but I hoped it could be improved. Generally I'm aiming for high precision and I'm perfectly okay with some of the noise left unclustered, so I thought about DBSCAN. However, using fixed DBSCAN parameters doesn't seem to be a good idea, because, if I understand correctly, larger dataset (as I said the size is not known in advance) will mean higher density → and probably broken clustering at some point (is this correct?). Therefore I have following questions:
Is it possible and reasonable to try to estimate DBSCAN parameters on the fly based on data size/density? If so, how can I approach this?
Is it a good idea at all to use DBSCAN in this setting? Would you recommend something more suitable?
Are there any general tips I should keep in mind for improving clustering precision? (That is, filtering out noise → getting separate "noise clusters" or "spam clusters" is also fine.)
Footnote: I'm well aware of topic modelling algorithms like LDA or NMF, but so far none of them worked well for me.