5
$\begingroup$

Why are hidden Markov models (HMM) called mixture models? What does it mix?

$\endgroup$
5
$\begingroup$

Mixture models are generic probability density functions which are the weighted sums of independent processes that add to a total density function with a total area of 1, which area is common to all probability density functions. Consider, for example that two people are cutting pencils on an assembly line. The first cuts a fraction $0<p<1$ of the pencils with an average pencil length of $\mu_1$ with a standard deviation of $\sigma_1$. A second person is cutting $1-p$ of the pencils with average pencil length of $\mu_2$ with a standard deviation of $\sigma_2$. Then the mixture distribution (normal distribution assumption) of pencils coming off of the assembly line is $MD(p,\mu_1,\sigma_1,\mu_2,\sigma_2)=pN(\mu_1,\sigma_1)+(1-p)N(\mu_2,\sigma_2)$.

In a hidden Markov model, the state (pencil cutters) is not directly visible, but the output (e.g., assembly line output), dependent on the state, is visible. Each state has a probability distribution over the possible output tokens ($p$ and $1-p$ in our case). Now a hidden Markov model does not have to be a mixture model, for example, it can be unimodal, but the mixture model type of hidden Markov model is simple to solve.

To better explore if, as claimed in Wikipedia, a hidden Markov model can be considered a generalization of a mixture model or whether that is just too narrow a view, I posed this as a separate question; Are there any examples of hidden Markov models that are not mixture models? And as it turns out convolutions can be HMM as well, and most people would consider convolution to be a different operation from mixture addition.

It would seem that HMM are not only useful for mixture models, but for convolution models and possibly others.

$\endgroup$
  • 5
    $\begingroup$ And to avoid any confusion: "Why are hidden Markov models (HMM) also called mixture models?" HMMs are NOT called mixture models. Mixture models and HMMs are 2 different models but as explained above, HMMs can embed mixture models as emission distributions. $\endgroup$ – Eskapp Dec 8 '16 at 16:04
  • 1
    $\begingroup$ @Eskapp,thanks ,as I read on wiki- [link] (en.wikipedia.org/wiki/Mixture_model) , mixture models are exactly what Carl explained and yes, it's only a part of HMM as u rightly said. $\endgroup$ – Arpit Sisodia Jan 28 '17 at 4:40
-2
$\begingroup$

(This answer would be better as a comment to build on @Eskapp's comment)

I think it is important to give the general and simple formula $$p(Y) = \sum_{X} p(X,Y) = \sum_{X} p(X)p(Y|X)$$ (also appearing on Wikipedia). This clearly shows that in HMM, it is the observation process ($Y$) which is modeled as a mixture.

However, as already noted, HMM are not called mixtures models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.