The problem in short:

I want to estimate (?) a lag-1 Markovian hidden process for offline multi-variate discrete-time time series with continuous distributions via smoothing, with no dimensionality reduction. The emission model is time invariant.

The data that I have:

I have a dataset of multivariate time series of $N = 100$ participants, each of $8$ variables (i.e. $Y_{t_{8 \times 1}} = [y^1_{t}, y^2_{t}, ..., y^8_{t}]'$) for $T=50$ time points, where I have around $5\%$ missing data. Missingness is most probably MCAR, but position and number of missig data differ from person to person.

The data is (rather highly) correlated ($Cov(Y_t)$ is absolutely non-diagonal) and the time series are autocorrelated but it drops significantly/sufficiently after lag-1.

What I need:

I have to find, for each participant, time series of $X_{t_{8 \times 1}} = [x^1_{t}, x^2_{t}, ..., x^8_{t}]'$ such that each $x^i_{t}$ is the lag-1 hidden Markov process of $y^i_{t}$, and it does not explain (or, is not contaminated by) any information of other $y^j_{t}$s.

I am skeptical that I can use HMM for this problem, at least since estimating the high number of parameters seems over-kill for this amount of data. So I assume Kalman filter would be a candidate here.

I have the following questions

  1. For this problem, do you suggest Kalman filter or something else?

  2. Does fitting a multivariate Kalman filter (or another model, based on your answer to the first question) prefered over multiple separate filters/mdels for each $y^j_{t}$? I think that it may help in estimating the distribution of $\eta_t$, $\epsilon_t$, and/or $\alpha_t$ (based on the notation in this answer.) However, as I said, I have to avoid the contamination mentioned above.

  3. Can/should I fit them separately to each participant, or I shoul/have to fit to all the data of all people? (and how?)

  4. Based on your answers so far, is this filtering/smoothing/model fitting feasible for the kind of data I have? I mean with this number of variables, data points, and missingness. I am in fact worried about overfitting or poor model fit.

  5. What R package/function can I use for this problem?

Thanks in advance!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.