I am clustering tweets which are related to eye fashion and they are extracted using keywords like mascara, eyeliner, eyeshadow, etc from twitter. I constructed a Tf-idf matrix (tweets x words) which was around (100000 x 550).

If I give this to K-Means and plot elbow plot I don't get a perfect elbow. (Its almost diagonal) But after applying truncated SVD with latent variables say 10 variables and then I do kmeans, I get a perfect elbow at 10.

I have tested this for various other numbers of latent variables too and I get perfect elbow at those numbers. Can the results be trusted in this case?

I don't see the clusters having similar topics. What is the best way to proceed in this case. I did try LDA but there also I dont see perfect distinction between topics. Is it because that the data itself is not clusterable?

  • $\begingroup$ This question can hardly be answered without the example: your data, your analysis, results, pics. $\endgroup$ – ttnphns Jun 24 '19 at 17:59
  • $\begingroup$ Tweets tend to not be clusterable with k-means well. They are too short and there is too much garbage on Twitter (and k-means does not tolerate noise well). Tough luck. $\endgroup$ – Has QUIT--Anony-Mousse Jun 30 '19 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.