I try to use GMM-HMM model to infer the topic of sentences in a short paragraph. While instead of using normal Baum-Welch optimization, I use viterbi training as follows.
- I use average of word vectors as the feature vector of a sentence
- Clustering the sentences by GMM. As a result, each sentence is assigned hard to a topic, which is the initial hidden state. And each topic is modeled by a Gaussian distribution.
- According to 2, I initialize the transition matrix by calculating the conditional probabilities of adjacent states in paragraphs. As for the emission probabilities, since each topic is modeled by a Gaussian, it can be computed directly.
- Now that the hmm parameters are initialized, I update the topics of sentences by Viterbi algorithm, and then update the Gaussian distributions according to the new topic assignment. Repeat these two steps until converge.
The problem is, the results of step 4 always lead to collapsed topic number. For example, initially I cluster the sentences using a 50-component GMM, while after viterbi training of step 4, only 2 states remained.
This result seems to be strange and I don't know why. I've checked my code many times and have not found anything wrong. This seems to be a reasonable result. Could any one give some explanations or ideas to do the right thing?