I am clustering a set of 50k products. I would expect the resulting clusters to be things like "organic chicken", "orange juice", etc. I am using the bag of words model, and there are about 8k features. I have tried using mini-batch k means, tuning the parameters including cluster size from 100 to 500. The algorithm produces some clusters which are extremely specific (which is good). However, it places the majority of products into about 2 clusters (which is very bad). I'm trying to investigate why the clusters are so uneven. It clearly has the ability to detect similar products based on similar words for some clusters, so I'm not sure why it places the majority into a small number of clusters. Maybe I should be using a different NLP model or a different clustering algorithm. Thanks!

  • 1
    $\begingroup$ What kind of stop word list are you using? Just a guess, I'd look at the words the big clusters share and see if any need to be on a stop word list. $\endgroup$ – roundsquare Jun 24 '17 at 15:16
  • $\begingroup$ I'm using the one that sklearn provides. I don't see any really common words amongst, i.e this really small sample [chocol sandwich cooki, all season salt, robust golden unsweeten oolong tea, green chile anytim sauc] taken from a really big cluster. Other clusters are really good such as [restaurant style bite size nacho tortilla chip, origin restaur style parti size tortilla chip, bite size tortilla chip]. $\endgroup$ – cosmosa Jun 24 '17 at 15:23
  • 2
    $\begingroup$ As others already said, clustering very short text documents is probably infeasible. However, as an off-topic tip, if possible, you could enrich your product instances with any kind of meta-data that could get (sales statistics, views/CTR counts, reviews, producer information, etc.). That could greatly help get to more sensible clusters, if available. $\endgroup$ – fnl Jun 26 '17 at 10:30

Text clustering barely works when you have long text. When you apply it too small text, it degenerates. Everything is similar to everything else, with no shades of grey to work with for clustering. Such data is simply too coarse.

Yes, you may be seeing some clusters that appear to be very good. Usually, they are caused by some rare word, and then all documents that contain this word. But you just cannot cluster the majority of documents this way. And the result is not at all like a k-means least squares result, but rather a keyword selection result. You might as well just take the least frequent words (but above some threshold) and then call everything that has this word (e.g. teriyaki) a "cluster". It does something, but is that "clustering"?

Bad news is, that I don't think there is an algorithm that will produce much better results. Because of the data being too coarse, the whole problem of clustering on such data is not well defined.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.