Why are hidden Markov models (HMM) called mixture models? What does it mix?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
7
Mixture models are generic probability density functions which are the weighted sums of independent processes that add to a total density function with a total area of 1, which area is common to all probability density functions. Consider, for example that two people are cutting pencils on an assembly line. The first cuts a fraction $0<p<1$ of the pencils with an average pencil length of $\mu_1$ with a standard deviation of $\sigma_1$. A second person is cutting $1-p$ of the pencils with average pencil length of $\mu_2$ with a standard deviation of $\sigma_2$. Then the mixture distribution (normal distribution assumption) of pencils coming off of the assembly line is $MD(p,\mu_1,\sigma_1,\mu_2,\sigma_2)=pN(\mu_1,\sigma_1)+(1-p)N(\mu_2,\sigma_2)$.
In a hidden Markov model, the state (pencil cutters) is not directly visible, but the output (e.g., assembly line output), dependent on the state, is visible. Each state has a probability distribution over the possible output tokens ($p$ and $1-p$ in our case). Now a hidden Markov model does not have to be a mixture model, for example, it can be unimodal, but the mixture model type of hidden Markov model is simple to solve.
To better explore if, as claimed in Wikipedia, a hidden Markov model can be considered a generalization of a mixture model or whether that is just too narrow a view, I posed this as a separate question; Are there any examples of hidden Markov models that are not mixture models? And as it turns out convolutions can be HMM as well, and most people would consider convolution to be a different operation from mixture addition.
It would seem that HMM are not only useful for mixture models, but for convolution models and possibly others.
-
6$\begingroup$ And to avoid any confusion: "Why are hidden Markov models (HMM) also called mixture models?" HMMs are NOT called mixture models. Mixture models and HMMs are 2 different models but as explained above, HMMs can embed mixture models as emission distributions. $\endgroup$– EskappDec 8, 2016 at 16:04
-
1$\begingroup$ @Eskapp,thanks ,as I read on wiki- [link] (en.wikipedia.org/wiki/Mixture_model) , mixture models are exactly what Carl explained and yes, it's only a part of HMM as u rightly said. $\endgroup$ Jan 28, 2017 at 4:40
-
$\begingroup$ Convolution is exactly the sum law of probability distributions. en.m.wikipedia.org/wiki/…. Does it mean it's exactly the same as the mixture addition? $\endgroup$ Sep 11 at 3:23
-
$\begingroup$ @BrianCannard Convolution of probability distributions, i.e., probability density functions, is equal to their convolution integral, e.g., see infinite support; Fourier transforms for one type of convolution integral. There are two others, real space convolution and Laplace transforms, the latter for semi-infinite support densities. $\endgroup$– CarlSep 11 at 6:10
-
1$\begingroup$ @BrianCannard More generally, density functions do not have to be probability density functions (pdf). In general, density functions add by convolution, and a mixture model is the scaled ordinary addition of density functions, which is not convolution. Convolution of density functions is not ordinary addition, mixture models are ordinary addition. It may be hard to grasp what convolution is, but trust me on this, it is not mixture modeling. $\endgroup$– CarlSep 11 at 6:21
$\begingroup$
$\endgroup$
(This answer would be better as a comment to build on @Eskapp's comment)
I think it is important to give the general and simple formula $$p(Y) = \sum_{X} p(X,Y) = \sum_{X} p(X)p(Y|X)$$ (also appearing on Wikipedia). This clearly shows that in HMM, it is the observation process ($Y$) which is modeled as a mixture.
However, as already noted, HMM are not called mixtures models.
