3
$\begingroup$

Mixture models lend themselves nicely to HMM (hidden Markov model) treatment. Obviously, HMM can be nonmixture or not resolve mixtures when there is only one resolvable input or the superposition of self-similar inputs. What I am asking for is whether or not there are any not so trivial examples of nonmixture HMM data, with the impact relating to an attempt to define HMM as non-trivially different than mixture model analysis, e.g., see https://stats.stackexchange.com/a/249712/99274 $\leftarrow$Let me ask, for example, if the example I gave is exactly correct or not.

Be mindful please that a hidden Markov model can be considered a generalization of a mixture model where the hidden variables (or latent variables), which control the mixture component to be selected for each observation, are related through a Markov process rather than independent of each other..

And moreover that In the standard type of hidden Markov model considered here, the state space of the hidden variables is discrete, while the observations themselves can either be discrete (typically generated from a categorical distribution) or continuous (typically from a Gaussian distribution).

$\endgroup$
9
  • 2
    $\begingroup$ I am not sure of your semantics, but do particle filters count? (The Wikipedia article for that one also appears a bit particular ... but many would include Kalman/particle filters as a type of HMM) $\endgroup$ – GeoMatt22 Feb 20 '17 at 23:03
  • $\begingroup$ @GeoMatt22 a particle filter is the algorithm, not the model. But I agree with the rest $\endgroup$ – Taylor Feb 21 '17 at 0:03
  • 2
    $\begingroup$ @Taylor that is fine. I was using "particle filter" as shorthand for something like "continuous-state HMM with no pre-defined parametric PDFs built in". Really just the idea that the class of state-space models/applications where particle filters are used must typically deal with multimodal state PDFs in practice (e.g. robotics). $\endgroup$ – GeoMatt22 Feb 21 '17 at 2:39
  • $\begingroup$ @GeoMatt22 Tried to fix semantics, let me know please. $\endgroup$ – Carl Feb 21 '17 at 15:31
  • 2
    $\begingroup$ Rabiner has a 1988 tutorial on HMMs in speech recognition that is widely read and cited to this day. He reviews several different types of HMMs. cs.cornell.edu/courses/cs481/2004fa/rabiner.pdf $\endgroup$ – Mike Hunter Feb 21 '17 at 15:41
2
$\begingroup$

This might deserve to be a comment:

One of the reasons I don't understand your question, or in particular your second sentence, is that I'm not sure unimodal is equivalent with non-mixture. I guess it would be nice to think of those as equivalent if the observed random variables were continuous and in particular Gaussian, but also even in that case, I bet if the locations of each of those Gaussians are close enough, or if the weights are in such a configuration, that one of the conditional distributions of the observations could be unimodal, still. Are we talking about the predictive distribution, or some other conditional distribution?

And it IS very different because the proportions are changing (this might be more of the answer you're looking for). Assuming a HMM is assuming a model for how the mixing proportions change (the probabilities, the locations, the scales, etc.). You're assuming they are a Markov Chain. A mixture model is a static mixture. A HMM is a dynamic mixture.

$\endgroup$
3
  • $\begingroup$ I frankly do not know if it helps because I do entirely understand some of what you are saying. Did you ask me to consider that, for example, the HMM model $f[N(X;\mu_1,\sigma_1),t]+g[N(X;\mu_2,\sigma_2),t]$, where $\int_{-\infty}^{\infty}f+gdt=1$ is somehow not a mixture model? Are you saying that a parametric generalization of a mixture model can be an HMM? Unimodal versus nonmixture and Gaussian, yup they seem used together in that context and I do not know about other contexts, any clarification appreciated. $\endgroup$ – Carl Feb 20 '17 at 22:44
  • $\begingroup$ @Carl no I don't recognize that. One thing I am saying is that there exist mixtures that are unimodal. $\endgroup$ – Taylor Feb 21 '17 at 0:07
  • $\begingroup$ Dumped the word unimodal, see if my changes make sense, please, BTW +1 for effort. $\endgroup$ – Carl Feb 21 '17 at 15:34
2
$\begingroup$

HMMs model ordered sequences of observations or time-series $X = \{x_1, x_2,..., x_T\}$, where each $x_i$ can be a discrete or continuous, uni-dimensional or multi-dimensional observation/data sample. The order of the samples, from $t = 1$ to $t=T$ is modeled by the HMM as well (via the transition matrix).

Mixture models cannot model any dynamics in the data. So HMM can be seen as a Dynamic Mixture Model as noted in this research report for instance that defines the HMMs in the context of Dirichlet mixture models (as their dynamic counterpart). The multimodality does not change anything to this, even when mixtures are not used, HMMs embed data dynamics while mixture models "see" data as unordered. (By the way, as someone commented, multimodal is a very confusing term I find in the context of HMMs...).

The only cases HMMs are similar to mixture models are if the number of states is 1 or, as said in the comments, if the transition matrix is the identity matrix added to the fact the initial state probability mass function is a vector of 0 with a unique 1, and that simple distributions and not mixtures are assigned to the sates. Otherwise, it is more a "collection" of mixtures. However, knowing that the transition matrix is the identity adds an information about the dynamism of the data, and I would see it as a "weak-equivalence". In the case of a mixture-based HMM, the mixing matrix also impacts making this "equivalence" even more challenged...

In general, a good introduction reading to HMMs is this paper by Rabiner and Juang who popularized their use in the speech processing community.

$\endgroup$
7
  • $\begingroup$ Do not understand as the observations can be either discrete or continuous. Yes, you can help. Put quotes in question to help clarify. $\endgroup$ – Carl Feb 20 '17 at 22:57
  • 1
    $\begingroup$ "So HMMs cannot be similar to mixture models except if the number of states is 1." This isn't true. You can have more than $1$ states along with an identity transition matrix $\endgroup$ – Taylor Feb 20 '17 at 23:25
  • $\begingroup$ Very true! I forgot this case :) Thank you for reminding me. I edit my answer $\endgroup$ – Eskapp Feb 21 '17 at 15:03
  • $\begingroup$ @Carl I added details and references. Hope it helps :) $\endgroup$ – Eskapp Feb 21 '17 at 15:18
  • 1
    $\begingroup$ @Eskapp +1 for effort and effect, look over semantic changes, see if I got that right or not, please. What about convolution? $\endgroup$ – Carl Feb 21 '17 at 15:36
1
$\begingroup$

With regrets, I did not want to answer my own question. What I was asking about was something other than mixture modelling. Here is one; Convolution HMM. @GeoMatt22, I had originally thought that "continuous-state HMM with no pre-defined parametric PDFs built in" was the most natural mixture model HMM definition, and did not even know about discrete modelling until it was mentioned here and I started looking into it a bit deeper. The general filter problem may be even more general than convolution as I suspect that its solution may be a general inverse solution, which is very general, indeed. But, the point here is that most people would probably agree that mixture models are distinct from convolution, and from the link, it would seem that there is such a thing as a convolution HMM. Indeed, a highly cited one, 600+. Having said that, I will award +1 to anyone who can (and has) helped with the semantics. What I am trying to do is come up with a short, good definition of HMM. And as @Geomatt22 says, the Wikipedia entry may be a bit 'particular' not to say wonky.

$\endgroup$
1
  • $\begingroup$ As far as I know, the most general definition of a hidden Markov model is a state-space model for an evolving state $x[t]$ (an arbitrary vector; can combine continuous and categorical components), whose dynamics satisfy 1) the Markov property, such that given the current state $x_t$, the (possibly stochastic) dynamics is independent of the history (i.e. $p[x_{t+1}|x_{0:t}]=p[x_{t+1}|x_t]$), and 2) whose state is at least partially hidden (i.e. some components are not observed). (Typically an HMM will be associated with a measurement function, cf. state-space-models wiki). $\endgroup$ – GeoMatt22 Feb 22 '17 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.