Mixture Model with dependant observations

Question

I am trying to model a process in which each datapoint is generated sequentially, so the current observation depends on the last one. Some example data could look like,

A = [4.5, 5, 6, 5.5, 8, 0, 0, 0, 0, 3, 2.3, 4.5] B = [4.5, 5, 6, 5.5, 8, 0, 0, 0, 0, 0, 0, 0] C = [0, 0, , 0, 0, 0, 0, 0, 0, 2.9, 2.2, 4.4]

the goal is to identify which objects have very similar observations. In the above case, A and B would be identified as being very similar, and A and C likewise.

My go-to solution was to fit a Gaussian Mixture Model, however the simple model does not take into account that the current observation depends on the last. This is also very evident from preliminary results, in which the following kind of combination is identified as being very similar,

B = [4.5, 5, 6, 5.5, 8, 0, 0, 0, 0, 0, 0, 0] D = [0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 3, 2]

and that should not be the case. And yes, I am aware that "being very similar" is a very vague definition, but I am asking this question mostly to get some ideas on how to go about the problem.

Is a mixture model even appropriate for this kind of analysis? If not, what are some alternatives? I think that it is really important any model takes into account the sequential nature of the data. Also,

If I missed any obvious previous posts on SE I apologise, I am still struggling a bit with the terminology and might have missed an obvious search term.

Cheers.

Update: Still trying to solve this problem. After some discussions on the Stan mailing list, I arrived at a model,

$$p(y_{2:T} | \theta, \mu, \sigma, beta) = \prod_{t \in 2:T} \sum_{k \in 1:K} \theta(k) \cdot \mathcal{N}\left(y(t) | \mu + \beta(k) \cdot y_{t-1}, \sigma\right) $$

So it has an autoregressive property. I'm not sure how well this model captures the signal, though. Early tests does not show very good results, so I'm not sure if the idea if worth following.

Another thought I got would be to calculate a similarity, or somehow encode the signal, so I can use a clustering algorithm instead. Any tips or suggestions on this?

Answer 1

I think you are looking for Hidden Markov Models. In particular see http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/tutorial%20on%20hmm%20and%20applications.pdf. In this paper on page 5, they pose 3 questions which they later answer. I think question 2 in particular seems (?) to be in line with what you are thinking

Answer 2

A sequential mixture model would look something like:

$\boldsymbol Y_i \sim \mathcal N_d(\boldsymbol \mu_{c_i}, \boldsymbol\Sigma_{c_i}) \ \ i=1,...,n$

$\boldsymbol c_i \sim \text{Multinom}(\boldsymbol p), \ \ i=1,...,n$

Where $\boldsymbol p$ is of length $k$, then:

$(\boldsymbol \mu_{c}, \boldsymbol\Sigma_{c}) \sim \text{Normal-Inverse-Wishart}(m,S,\boldsymbol \Psi, v), \ \ c=1,...,k$

Now the important thing for introducing dependency will be making $\boldsymbol \Psi$ have a suitable structure. E.g. precision matrix of an AR1 process is tri-diagonal (e.g. see here). Generally the IW prior will capture an arbitrary dependency structure if $\boldsymbol \Psi$ is only diagonal, but this might help it along.

This is different than a Hidden Markov Model. The difference is that mixture models suppose the whole series is sampled at once from a particular type of process, but you don't know which process. HMMs suppose that at each time point $\boldsymbol Y_t$ is sampled from a different distribution, and the distribution changes over time via a Markov chain.

Looking at the samples of data the HMM might be interesting, as it looks as though there are bursts of similar behaviour, followed by changes.

When you say you used a GMM previously, was this multivariate (the whole time series at once) or univariate (pooling observations across series and time)?