I started working on a topic definition task and my initial approach was as follows:
- Use LDA (Latent Dirichlet Allocation) to obtain the initial topic distribution for each of my documents.
- Then use another clustering technique (like k-means) over the topic distributions to find the "real" clusters that define each document.
My logic was that each document actually has more than one topic assigned; some trend heavily to a single topic but others seem to be a mix of many topics (this does not change if I increase the number of topics and I just end up with worse or almost repeated ones).
Nevertheless, all the papers I've read just use the dominant topic as the classifier for each document, even if it was only slightly dominant. I have been reading answers here and some said that it did not make sense to use K-means on the LDA output vector and I truly don't see why, I still have an unlabeled dataset and I need to find the structure of it, so LDA is only a dimensionality reduction step for me and the final classification would be with another algorithm that gives me a hard class.
I understand that if everyone is just taking the first topic there must be a valid reason, but I can't see it at the moment and I would really like to understand the flaw in what I wanted to do.