# Why are hidden Markov models considered 'mixture models'?

I'm a bit confused by the conception of "mixture model" I'm studying hidden Markov model, which is frequently referred to as a "mixture model". But I don't know what the term "mixture" implies.

Suppose the HMM has state variable sequence $v_1,v_2,\dots,v_t$, $v_i\in\{1,\dots,k\}$. The emission model has pdf $f(y_i\mid\theta_{v_i})$. For each state $m$, the emission probability is a gaussian mixture $f(y\mid\theta_m)=\frac{1}{3}\mathcal{N}(\mu_m,\sigma^2)+\frac{1}{3}\mathcal{N}(\mu_m-10,\sigma^2)+\frac{1}{3}\mathcal{N}(\mu_m+20,\sigma^2)$

Why is it a mixture model? Is it because it has multiple states and each state has its own emission probability? Or is it because each state uses a Gaussian mixture?

According to Wikipedia, the second understanding is more probable. However, intuitively, I think the first understanding is more reasonable.

I know the definition of mixture model in Wikipedia. I (think I) know what is a mixture model. Now I need to know what is not a mixture model.

Is a multi-state model a mixture model in general?

Another evidence to support the first understanding. In the first three paragraphs of section 7, the paper claims

the observation $y_{t+1}$ is drawn from the mixture component indexed by $v_{t+1}$.

Here, $v_{t+1}\in\{1,\dots,k\}$ is a state variable, and given the realization of the variable, $f(y_{t+1}\mid\theta_{v_{t+1}})$ is an emission probability. Hence this multi-state model is called a "mixture model" and each probability (corresponding to a state), without being summed up, is called a "mixture component" in the paper.

Moreover, there might be a way to "mediate" the two understandings: if each state $m$ has an occurrence probability $\pi_m$, then, the marginal emission probability (state variable is integrated) is $\sum_{m=1}^k\pi_mf(y\mid\theta_m)$, this is obviously a mixture model.

Hence we have a general claim: a multi-state distribution can also be called a mixture model due to the above transformation.

Is this claim valid? Why?

A mixture distribution is a distribution made up of components which are themselves other distributions. The distribution is a mixture of its components.

The simplest case is the finite mixture, of which you give an example in your question.

$f(x) = p_1 f_1(x) + p_2 f_2(x) + ... + p_k f_k(x)$

$F(x) = p_1 F_1(x) + p_2 F_2(x) + ... + p_k F_k(x)$

where the mixing proportions $p_i$ sum to 1.

You can conceive of a mixture as first choosing a component with probability $p_i$ and then being generated from that component (indeed, that's a way to simulate from mixture distributions).

Here's the mixture you describe (with $\mu_m=50$ and $\sigma=5$):

Is a multi-state model a mixture model in general?

That depends on what you're looking at/how you frame it. If you look at the unconditional state (unconditional on earlier time), then it could be framed as a finite mixture of degenerate distributions (representing the states).

The paper itself states it quite explicitly (if framed slightly differently), right in the middle of the three paragraphs you pointed to:

This is essentially a dynamic variant of a finite mixture model, in which there is one mixture component corresponding to each value of the multinomial state.

• So each $f_i(x)$ has the same support or domain. Hence the multi-state part does not contribute to "mixture model". – user3813057 Jul 13 '15 at 2:45
• In that example they have the same support, but that's not a requirement; I could have a mixture of a U(0,1) and a U(4,6) for example. And you can mix discrete and continuous variables. – Glen_b Jul 13 '15 at 3:37
• Thanks for your answer. Your answer explained what is a mixture model, but didn't say clearly what is not a mixture model. I've edited my question, where I found a "non-orthodox mixture model" according to your answer can be "transformed" into a "orthodox mixture model". Can you please answer the modified question? – user3813057 Jul 14 '15 at 5:39
• So, even though the state conditional probability itself is not a mixture model, the model becomes a mixture model when the state variable is integrated out. Hence, a multi-state model can be transformed into a mixture model form although the original form is not a mixture model. Therefore, it's acceptable to call a multi-state model a (non-orthodox) mixture model. – user3813057 Jul 14 '15 at 16:43
• I don't agree with your conclusion at all. – Glen_b Jul 15 '15 at 0:24

A mixture model (MM) is a probabilistic model which represents subpopulations within the whole population, therefore it will approximate more closely data that is made up of two or more groups that have different individual distributions, than a model that assumes all data comes from one unique distribution.

In your HMM example, each state's emission probability is given by a gaussian mixture, which is the "mixture" part of the model.

This does not mean that all states have the same emission probabilities, since for each state the mean of each gaussian is different, hence the $\mu_m$ for each state m

• This explanation aggrees with the definition. However, some work calls multi-state as a mixture model. – user3813057 Jul 13 '15 at 2:49
• After briefly reviewing the linked work I'm not sure it effectively states that... but anyway, not all multi-state models are mixture models, and in your example it (as in the linked paper IMO) the "mixture model" name comes from dealing with a mixture of components in the probability distributions. – twalbaum Jul 13 '15 at 3:37
• I'm still not clear which part can be called "mixture", and which part cannot. In formula (10) of the paper mentioned above, given the state label $z_i$ the observed variable $x_i$ has distribution $F(x_i\mid\phi_{z_i})$. Since each state $k$ has appearance probability $\pi_k$, the marginal distribution of $x_i$ can be represented as $X_i\sim\sum_{k=1}^\infty\pi_kF(x_i\mid\phi_k)$. Therefore, a multi-state model is transformed to a multi-component model, and hence a mixture model. In this case, can we get a general claim that a multi-state model is also a mixture model due to the transforming? – user3813057 Jul 14 '15 at 0:44
• I think the bottom line here is you should not use the term mixture model to refer to any multi-state model. Whether a transformation makes one become a mixture model is another story... – twalbaum Jul 14 '15 at 1:14