I have a dataset of short message conversations (from 1 to 20 words). I would like to cluster the messages that were sent to me to extract the different topics that were discussed by my interlocutors.
I have tried using a FasText representation for my messages where the message representation is obtained by averaging the individual word representations.
After that, I have clustered using the HDBSCAN algorithm. The results I obtained were not satisfactory as most of the points were assigned to the noise cluster.
Examples from my dataset (translated):
- "Good Morning" ==> Greeting
- "Are you available tomorrow for a call" ==> Availability
- "My phone number is 0123456" ==> Contact details
I was wondering whether there would be a better way to represent my messages: Tf-Idf matrix factorization? Alternatively, a clustering algorithm with different assumptions and that scales well? The reason why I chose HDBSCAN was that the data didn't exhibit a globular structure when visualized using the t-SNE algorithm (k-means wouldn't work).