# Is a GMM-HMM equivalent to a no-mixture HMM enriched with more states?

I'm trying to model sequence data that has 5 hidden states. Observation data conditional to each state is gaussian except for one state for which mixture of 2 gaussians seems more appropriate. Unfortunately, the R package that I'm using (depmix) does not seem to support (without extending the package) a GMM as a possible response distribution. So I was considering the possibility of adding a 6th state so I could interpret one of this enriched set of states as a state for which observation distribution is the first gaussian in my above mixture and another one as a state for which observation distribution is the second gaussian.

Am I wrong thinking that the two approaches are equivalent?

It is not exactly equivalent: the 6-state HMM can model everything the GMM-HMM can, but not the other way around.

Suppose you start with the GMM-HMM, with $$s_5$$ being the GMM state, and turn it into the 6-state HMM with states $$s_6$$ and $$s_7$$ instead of $$s_5$$.

Let $$p_6$$ and $$p_7$$ be the prior probabilities of the two components of the GMM (that are then transformed into states $$s_6$$ and $$s_7$$).

For every transition from a state $$s_i$$ to $$s_5$$ in the GMM-HMM (with probability $$t$$), create two transition probabilities in the 6-state HMM:

• $$s_i$$ to $$s_6$$ with probability $$t \cdot p_6$$
• $$s_i$$ to $$s_7$$ with probability $$t \cdot p_7$$

For every transition from $$s_5$$ to a state $$s_i$$ in the GMM-HMM (with probability $$t$$), create two transition probabilities, respectively from $$s_6$$ and $$s_7$$, going to $$s_i$$, both with the same probability $$t$$.

If I am not mistaken, the resulting 6-state HMM is equivalent to the GMM-HMM.

However, the other way around doesn't always work. Imagine you are starting the the 6-state HMM.

Suppose that the transition probabilities for $$s_i \rightarrow s_6$$ and $$s_i \rightarrow s_7$$ are not equal do not have the same ratio as $$s_j \rightarrow s_6$$ and $$s_j \rightarrow s_7$$ (EDIT). You could not carry this information into the GMM-HMM.

In short, the 6-state HMM should be able to represent everything the GMM-HMM can, and more.

• I'm not sure that last "suppose" is correct. What if the mixture probabilities are, say, 0.3 and 0.7? Wouldn't that imply that the transition probabilities into $s_6$ and $s_7$ weren't equal? – jbowman Sep 26 '18 at 19:44
• On the other hand, there is an implication of the GMM that $p_{i,6}/p_{i,7} = p_{j,6}/p_{j,7}$ for all $i,j$, well except the zero transition probabilities of course, which doesn't have to be enforced by the 6-state HMM model, so your fundamental point is correct. – jbowman Sep 26 '18 at 19:45
• @jbowman you are correct, I've changed it. – Vincent B. Lortie Sep 26 '18 at 20:05
• @VincentB.Lortie so should we conclude that in this case the mixture is better because it's a more parcimonious model? – Patrick Sep 27 '18 at 13:12
• First, I would look into how well both fit the data. If the GMM-HMM is really sufficient to model the data, then it should perform comparably to the 6-state HMM, or even better if the 6-state HMM overfits. Then, I suppose you could make a case that the GMM-HMM has fewer parameters than the HMM. – Vincent B. Lortie Sep 27 '18 at 13:24

No you are not wrong thinking that.

If $$Y \mid X_1 \sim \alpha f_1(y) + (1-\alpha)f_2(y)$$, then you can also let $$X_2 \sim \text{Bernoulli}(\alpha)$$ independently and say $$Y \mid X_1, X_2 = 1 \sim f_1(y)$$ and $$Y \mid X_1, X_2 = 0 \sim f_2(y).$$ This is because $$f_{Y|X_1}(y \mid x_1) = \sum_{i=1}^2f_{Y|X_1,X_2}(y \mid x_1, x_2) f(x_2) = \alpha f_1(y) + (1-\alpha)f_2(y).$$ Keep in mind the sequence through time $$\{X_2^t\}_t$$ is iid, and so the Markov structure is overkill (but still perfectly fine).

• could you clarify what do you mean by "Markov structure is overkill"? – Patrick Sep 26 '18 at 20:19
• @Patrick I mean iid random sample data are less complicated to analyze than a Markov chain because each element does not rely on the chain's past. If you wanted to think of $\{X_{2}^t\}$ as a two-state Markov chain, then its transition matrix would have columns with all the same elements. – Taylor Sep 26 '18 at 20:25