0
$\begingroup$

I have a dataset of short message conversations (from 1 to 20 words). I would like to cluster the messages that were sent to me to extract the different topics that were discussed by my interlocutors.

I have tried using a FasText representation for my messages where the message representation is obtained by averaging the individual word representations.

After that, I have clustered using the HDBSCAN algorithm. The results I obtained were not satisfactory as most of the points were assigned to the noise cluster.

Examples from my dataset (translated):
- "Good Morning" ==> Greeting
- "Are you available tomorrow for a call" ==> Availability
- "My phone number is 0123456" ==> Contact details

I was wondering whether there would be a better way to represent my messages: Tf-Idf matrix factorization? Alternatively, a clustering algorithm with different assumptions and that scales well? The reason why I chose HDBSCAN was that the data didn't exhibit a globular structure when visualized using the t-SNE algorithm (k-means wouldn't work).

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ How about labeling some of them manually and then try some supervised or semi-supervised learning method? $\endgroup$
    – Jan Kukacka
    Commented Oct 1, 2018 at 6:47
  • $\begingroup$ This could be an option. My issue is that I don't know the number of different topics I have in my data hence, any labeling might miss some labels. Let's assume I want to follow the fully unsupervised approach, would you say that the message vectors I am using make sense or are there better options in your opinion? $\endgroup$
    – ryuzakinho
    Commented Oct 1, 2018 at 6:52
  • $\begingroup$ I don't have much experience with NLP so I can't say if your representation is good. Anyway, you can always start with fewer labels and if you feel you missed something, just add a new category and train your model from scratch. $\endgroup$
    – Jan Kukacka
    Commented Oct 1, 2018 at 7:50

1 Answer 1

Reset to default
1
$\begingroup$

If you have some confidence that your data is not that noisy it could be that the manifold structure in high dimensions is spreading out your density too much for HDBSCAN. You can try reducing down to fewer dimensions, either with something like TruncatedSVD, or UMAP (the docs provide an example of this). Be warned that UMAP will tend to collect points together, so if you have noise it will suck it into clusters. Further the results are now quite dependent on the dimension reduction hyper-parameters, so care should be taken. Approaches like this are not necessarily the first option you should take -- you run the risk of distorting the cluster structure with the dimension reduction -- but it can work, and if you have the time to try it and hand validate some of the clusters, it might be what you need.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. I have tried using dimension reduction based on this approach: (github.com/vyraun/Half-Size). Then, I tried using UMAP but, I wanted first to fit it on the .vec file like the approach above but I ended up with a huge model (around 50GB) of ram. Would you recommend fitting UMAP just on my data and not the actual FastText vec files. $\endgroup$
    – ryuzakinho
    Commented Oct 24, 2018 at 18:22
  • $\begingroup$ I would try working on just your data first -- that seems a more reasonable approach than trying to embed the entire vocabulary. $\endgroup$
    – Leland McInnes
    Commented Oct 24, 2018 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.