# Mixture Model with dependant observations

I am trying to model a process in which each datapoint is generated sequentially, so the current observation depends on the last one. Some example data could look like,

 A = [4.5, 5, 6, 5.5, 8, 0, 0, 0, 0, 3, 2.3, 4.5] B = [4.5, 5, 6, 5.5, 8, 0, 0, 0, 0, 0, 0, 0] C = [0, 0, , 0, 0, 0, 0, 0, 0, 2.9, 2.2, 4.4] 

the goal is to identify which objects have very similar observations. In the above case, A and B would be identified as being very similar, and A and C likewise.

My go-to solution was to fit a Gaussian Mixture Model, however the simple model does not take into account that the current observation depends on the last. This is also very evident from preliminary results, in which the following kind of combination is identified as being very similar,

 B = [4.5, 5, 6, 5.5, 8, 0, 0, 0, 0, 0, 0, 0] D = [0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 3, 2] 

and that should not be the case. And yes, I am aware that "being very similar" is a very vague definition, but I am asking this question mostly to get some ideas on how to go about the problem.

Is a mixture model even appropriate for this kind of analysis? If not, what are some alternatives? I think that it is really important any model takes into account the sequential nature of the data. Also,

If I missed any obvious previous posts on SE I apologise, I am still struggling a bit with the terminology and might have missed an obvious search term.

Cheers.

Update: Still trying to solve this problem. After some discussions on the Stan mailing list, I arrived at a model,

$$p(y_{2:T} | \theta, \mu, \sigma, beta) = \prod_{t \in 2:T} \sum_{k \in 1:K} \theta(k) \cdot \mathcal{N}\left(y(t) | \mu + \beta(k) \cdot y_{t-1}, \sigma\right)$$

So it has an autoregressive property. I'm not sure how well this model captures the signal, though. Early tests does not show very good results, so I'm not sure if the idea if worth following.

Another thought I got would be to calculate a similarity, or somehow encode the signal, so I can use a clustering algorithm instead. Any tips or suggestions on this?

I think you are looking for Hidden Markov Models. In particular see http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/tutorial%20on%20hmm%20and%20applications.pdf. In this paper on page 5, they pose 3 questions which they later answer. I think question 2 in particular seems (?) to be in line with what you are thinking

A sequential mixture model would look something like:

$\boldsymbol Y_i \sim \mathcal N_d(\boldsymbol \mu_{c_i}, \boldsymbol\Sigma_{c_i}) \ \ i=1,...,n$

$\boldsymbol c_i \sim \text{Multinom}(\boldsymbol p), \ \ i=1,...,n$

Where $\boldsymbol p$ is of length $k$, then:

$(\boldsymbol \mu_{c}, \boldsymbol\Sigma_{c}) \sim \text{Normal-Inverse-Wishart}(m,S,\boldsymbol \Psi, v), \ \ c=1,...,k$

Now the important thing for introducing dependency will be making $\boldsymbol \Psi$ have a suitable structure. E.g. precision matrix of an AR1 process is tri-diagonal (e.g. see here). Generally the IW prior will capture an arbitrary dependency structure if $\boldsymbol \Psi$ is only diagonal, but this might help it along.

This is different than a Hidden Markov Model. The difference is that mixture models suppose the whole series is sampled at once from a particular type of process, but you don't know which process. HMMs suppose that at each time point $\boldsymbol Y_t$ is sampled from a different distribution, and the distribution changes over time via a Markov chain.

Looking at the samples of data the HMM might be interesting, as it looks as though there are bursts of similar behaviour, followed by changes.

When you say you used a GMM previously, was this multivariate (the whole time series at once) or univariate (pooling observations across series and time)?

• Interesting. Do you have any core literature on those kind of mixture models? A quick google search did not provide anything really useful. – demodw May 13 '15 at 9:36
• I forgot to add, When we fit the GMM previously, we just used all of the data without any pooling. I am not sure if it was fit using a multivariate or univariate. We used the variational implementration of a GMM in sklearn (scikit-learn.org/stable/modules/dp-derivation.html) – demodw May 13 '15 at 10:45
• At a first glance the docs for that package look to me as though they treat each observation as independent once the mixture component is fixed. – conjectures May 13 '15 at 10:53
• If you look up Dirichlet process mixture models you'll find some more explanation, but in the setting of not knowing how many components to use. Also looking up conjugate inference for multivariate normal with unknown mean and covariance. – conjectures May 13 '15 at 10:55
• By chance, do you know of any software implementing a model like the one you described above with Normal-Inverse-Wishart? I tried skimming over the literature, but using DP-MM or conjugate inference for multivariate normal as keywords did not really yield anything. – demodw May 18 '15 at 11:22