How should longitudinal data be inputted into a HMMmodel (I don't care if the package is seqlearn, hmmlearn, pomegranate,...)? All these packages don't have a proper documentation on how to input data where for each timestep there are multiple variables. Considering the change of these specific variables over X timesteps it should be able to predict if it as a 1 or a 0 and it should be able to learn it by itself. I read that seqlearn is the best choice for this binary classification problem.

Until know I have seen formattings like this passing by:

X1 = [1, 2, 0, 1, 1]
X2 = [42, 42]

where you have two 1D sequences and afterwards you can do:

X = np.append(X1, X2)
lengths = [len(X1), len(X2)]
X = np.array([1, 1, 0, -1, -1])
model = hmm.GaussianHMM(n_components=2, n_iter=100)

I don't succeed in formatting my longitudinal data in such a way that it can be read in because the example above is not panel data. How should I arrange the X1, X2,...


My view is that the data for an HMM should be stacked up by time and hierarchical cross-section to produce a sensible model. Consider a simple example from an educational HMM study:

  • a single variable for time which can be very granular, e.g., daily hours and seconds

  • a variable representing the smallest unit of analysis, e.g., this could be the student

  • various higher order variables as appropriate, e.g., class or teacher, school, district, etc.

The variable for time can be aggregated into new variables such as hour of day, day of week, month, quarter, year, and so on, as appropriate to capture temporal seasonalities.

Here is a spreadsheet-type representation of the structure of this information:

enter image description here

This data structure enables the HMM to capture changes in the various aspects of the cross-sectional and temporal information.

Note that this structure is completely different from the structures associated with many machine learning algorithms which all too frequently destroy structure and variance, e.g., image-mining analyses which transform a digitized image into a single record of pixel-level information.

| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.