I am clustering a set of 50k products. I would expect the resulting clusters to be things like "organic chicken", "orange juice", etc. I am using the bag of words model, and there are about 8k features. I have tried using mini-batch k means, tuning the parameters including cluster size from 100 to 500. The algorithm produces some clusters which are extremely specific (which is good). However, it places the majority of products into about 2 clusters (which is very bad). I'm trying to investigate why the clusters are so uneven. It clearly has the ability to detect similar products based on similar words for some clusters, so I'm not sure why it places the majority into a small number of clusters. Maybe I should be using a different NLP model or a different clustering algorithm. Thanks!
Text clustering barely works when you have long text. When you apply it too small text, it degenerates. Everything is similar to everything else, with no shades of grey to work with for clustering. Such data is simply too coarse.
Yes, you may be seeing some clusters that appear to be very good. Usually, they are caused by some rare word, and then all documents that contain this word. But you just cannot cluster the majority of documents this way. And the result is not at all like a k-means least squares result, but rather a keyword selection result. You might as well just take the least frequent words (but above some threshold) and then call everything that has this word (e.g. teriyaki) a "cluster". It does something, but is that "clustering"?
Bad news is, that I don't think there is an algorithm that will produce much better results. Because of the data being too coarse, the whole problem of clustering on such data is not well defined.