# How to model multiple multivariate time series for binary classification?

The data consist of repeated measurements of multiple individuals, where more variables are measured at each time point. Individuals fall into mutually exclusive groups. Number of measurements may differ with respect to individual, i.e some individuals have longer and some have shorter history. The the grid of time points is equidistant. Each observation has a binary label, typically long history of 0-s and some of them then switch to 1. How to predict the probability of the label being 1 in next step given the history? Ideally the model should be interpretable, i.e. if I say there is 80% chance that the individual A1 will become 1 in next time step and 60% chance it will become 1 in second step given it did not become 1 in first forward step, I would like to have some explanation why there is this chance...

I will use the notation $$y_k(t)$$ to refer to the observation at time $$t$$ for the $$k$$-th individual.

The simplest solution I can see, if we assume little to no measurement error, would be to develop 2 states Markov Chain model with probabilities $$p_{ij} = P\big( y(t + 1) = j \, | \, y(t) = i \big)$$ such as $$p_{00} + p_{01} = 1$$ and $$p_{10} +p_{11} = 1$$. You can further add individual-dependence by specifying mixed effects on $$p_{10}$$ and $$p_{01}$$, for example with $$\operatorname{logit}(p_{01}^{(k)}) \sim \mathcal{N}(\mu_{01}, \sigma_{01}^2)$$ and $$\operatorname{logit}(p_{10}^{(k)}) \sim \mathcal{N}(\mu_{10}, \sigma_{10}^2)$$. If we are interested in higher order lags (e.g. $$t-2$$ and $$t-1$$ to predict $$t$$) we can simply increase the number of states in the Markov Chain.

This model can be implemented with a standard logistic regression. In R, if you have a dataframe df with columns y (observation at $$t$$), y_lag (observation at $$t-1$$), and k, you could fit this with lme4::glmer(y ~ (y_lag + 1 | k), family = "binomial"). The parametrisation would be different than the one described above, as the intercept coefficient would correspond to $$\operatorname{logit}(p_{01})$$ and the intercept + slope would correspond to $$\operatorname{logit}(p_{11})$$.

An alternative would be to develop a bespoke Bayesian model using the probabilistic programming language Stan for example, this is usually what I do in my work. For example, if we assume measurement errors, we can introduce the latent variable $$p_k(t) = P(y_k(t) = 1)$$ and assume a mixed effect autoregressive model on $$\operatorname{logit}p_k(t)$$ (taking the logit ensures that we are in $$\mathbb{R}$$). For instance, for a first order autoregressive model: $$\operatorname{logit}p_k(t + 1) \sim \mathcal{N}\big(\alpha_k\operatorname{logit}p_k(t) + b_k , \sigma^2\big)$$, where $$\alpha_k$$ and $$b_k$$ are the individual-dependent autocorrelation parameter and intercept, respectively. Finally, we can specify a mixed effect on $$\alpha_k \sim \mathcal{N}(\mu_\alpha, \sigma_\alpha^2)$$ and $$b_k \sim \mathcal{N}(\mu_b, \sigma_b^2)$$. With a Bayesian model, we would need set priors but then this is context-specific so I cannot address this here.

• Hi, thanks for the great idea! I have some additional questions (I am not very good in bayesian statistics, but I really love the idea): how can I be sure that gaussian distribution is correct for my logits? Is it possible to incorporate some similarities between individuals (or is this the part of the mixed ef. model) ? Is this model OK if variables are of different types (binary, categorical, continuous, discrete nominal, ratios, percentages) ? Eventually could you please point me to some literature? Thanks again :) Commented Oct 14, 2020 at 7:38
• I edited my answer because the Markov Chain can actually be fitted using a standard logistic regression, so you may not need to go Bayesian. The Gaussian distribution for the mixed effect is an assumption, it is usually the default and can be computationally convenient, but you could for example replace it by a Student t distribution if you consider that some individuals are outliers. The mixed effects that the individuals are similar as the individual-dependent parameters are pooled toward the common mean (the $\mu$s). Commented Oct 14, 2020 at 8:18
• The Markov Chain approach is only suitable for binary or categorical data, but the Bayesian approach with a latent variables could be extended to other kinds of variables. Here, we assume the measurement distribution is a Bernoulli, for continuous variables we could assume it is a Gaussian, for count data a Poisson and for ratio/percentage a Beta distribution. Commented Oct 14, 2020 at 8:21
• My understanding is that $y$ is one of the $x_1(t), ..., x_p(t)$. What you include in the model depends on what you are trying to achieve, but I would think it makes more sense to only include lags. If $y=x_1$, it would look like this: lme4::glmer(x1 ~ (x1_lag + ... xp_lag + 1 | k), family = "binomial"). It is still a Markov Chain, but with $2^p$ states (each state is defined by the tuple $(x_1, ..., x_p)$. Note that the other alternative extends naturally to the multivariate case by replacing the normal distribution of the latent model by a multivariate normal distribution. Commented Oct 14, 2020 at 9:20
• Yes, this should be possible if you can develop a univariate model for each of the $x$s. I think the main challenge would lie in specifying the priors, as the latent variables for $x_1$ and $x_2$ may have different scales. Commented Oct 14, 2020 at 12:18