4
$\begingroup$

I'm trying to model sequence data that has 5 hidden states. Observation data conditional to each state is gaussian except for one state for which mixture of 2 gaussians seems more appropriate. Unfortunately, the R package that I'm using (depmix) does not seem to support (without extending the package) a GMM as a possible response distribution. So I was considering the possibility of adding a 6th state so I could interpret one of this enriched set of states as a state for which observation distribution is the first gaussian in my above mixture and another one as a state for which observation distribution is the second gaussian.

Am I wrong thinking that the two approaches are equivalent?

$\endgroup$
4
$\begingroup$

It is not exactly equivalent: the 6-state HMM can model everything the GMM-HMM can, but not the other way around.

Suppose you start with the GMM-HMM, with $s_5$ being the GMM state, and turn it into the 6-state HMM with states $s_6$ and $s_7$ instead of $s_5$.

Let $p_6$ and $p_7$ be the prior probabilities of the two components of the GMM (that are then transformed into states $s_6$ and $s_7$).

For every transition from a state $s_i$ to $s_5$ in the GMM-HMM (with probability $t$), create two transition probabilities in the 6-state HMM:

  • $s_i$ to $s_6$ with probability $t \cdot p_6$
  • $s_i$ to $s_7$ with probability $t \cdot p_7$

For every transition from $s_5$ to a state $s_i$ in the GMM-HMM (with probability $t$), create two transition probabilities, respectively from $s_6$ and $s_7$, going to $s_i$, both with the same probability $t$.

If I am not mistaken, the resulting 6-state HMM is equivalent to the GMM-HMM.

However, the other way around doesn't always work. Imagine you are starting the the 6-state HMM.

Suppose that the transition probabilities for $s_i \rightarrow s_6$ and $s_i \rightarrow s_7$ are not equal do not have the same ratio as $s_j \rightarrow s_6$ and $s_j \rightarrow s_7$ (EDIT). You could not carry this information into the GMM-HMM.

In short, the 6-state HMM should be able to represent everything the GMM-HMM can, and more.

$\endgroup$
  • 1
    $\begingroup$ I'm not sure that last "suppose" is correct. What if the mixture probabilities are, say, 0.3 and 0.7? Wouldn't that imply that the transition probabilities into $s_6$ and $s_7$ weren't equal? $\endgroup$ – jbowman Sep 26 '18 at 19:44
  • 1
    $\begingroup$ On the other hand, there is an implication of the GMM that $p_{i,6}/p_{i,7} = p_{j,6}/p_{j,7}$ for all $i,j$, well except the zero transition probabilities of course, which doesn't have to be enforced by the 6-state HMM model, so your fundamental point is correct. $\endgroup$ – jbowman Sep 26 '18 at 19:45
  • $\begingroup$ @jbowman you are correct, I've changed it. $\endgroup$ – Vincent B. Lortie Sep 26 '18 at 20:05
  • $\begingroup$ @VincentB.Lortie so should we conclude that in this case the mixture is better because it's a more parcimonious model? $\endgroup$ – Patrick Sep 27 '18 at 13:12
  • $\begingroup$ First, I would look into how well both fit the data. If the GMM-HMM is really sufficient to model the data, then it should perform comparably to the 6-state HMM, or even better if the 6-state HMM overfits. Then, I suppose you could make a case that the GMM-HMM has fewer parameters than the HMM. $\endgroup$ – Vincent B. Lortie Sep 27 '18 at 13:24
4
$\begingroup$

No you are not wrong thinking that.

If $Y \mid X_1 \sim \alpha f_1(y) + (1-\alpha)f_2(y)$, then you can also let $X_2 \sim \text{Bernoulli}(\alpha)$ independently and say $$ Y \mid X_1, X_2 = 1 \sim f_1(y) $$ and $$ Y \mid X_1, X_2 = 0 \sim f_2(y). $$ This is because $$ f_{Y|X_1}(y \mid x_1) = \sum_{i=1}^2f_{Y|X_1,X_2}(y \mid x_1, x_2) f(x_2) = \alpha f_1(y) + (1-\alpha)f_2(y). $$ Keep in mind the sequence through time $\{X_2^t\}_t$ is iid, and so the Markov structure is overkill (but still perfectly fine).

$\endgroup$
  • $\begingroup$ could you clarify what do you mean by "Markov structure is overkill"? $\endgroup$ – Patrick Sep 26 '18 at 20:19
  • $\begingroup$ @Patrick I mean iid random sample data are less complicated to analyze than a Markov chain because each element does not rely on the chain's past. If you wanted to think of $\{X_{2}^t\}$ as a two-state Markov chain, then its transition matrix would have columns with all the same elements. $\endgroup$ – Taylor Sep 26 '18 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.