How to model longitudinal big data?

Question

Traditionally we use mixed model to model longitudinal data, i.e. data like:

id obs age treatment_lvl yield
1  0   11   M  0.2
1  1   11.5 M  0.5
1  2   12   L  0.6
2  0   17   H  1.2
2  1   18   M  0.9

we can assume random intercept or slope for different persons. However the question I'm trying to solve will involve huge datasets (millions of persons, 1 month daily observation, i.e. each person will have 30 observations), currently I'm not aware if there are packages can do this level of data.

I have access to spark/mahout, but they do not offer mixed models, my question is, is there anyway that I can modify my data so that I can use RandomForest or SVM to model this dataset?

Any feature engineering technique I can leverage on so that it can help RF/SVM to account for auto-correlation?

Many thanks!

Some potential methods but I could not afford the time to write them into spark

How can I include random effects into a randomForest

SVM regression with longitudinal data

Answer 1

If you only have a few variables, like in the example, then you should have no problem with some variant of lme4.

Where machine learning techniques really shine is when you've got a lot of variables and you wish to model nonlinearities and interactions between your variables. Few ML approaches have been developed that can do this with longitudinal data. RNNs are one option, though these are generally optimized for time series problems, rather than panel data.

In principle, a feed-forward neural network is a (generalized) linear model, with regressors that are nonlinear functions of the input data. If the derived regressors -- the top layer of the model before the output -- are considered the nonparametric part, then there is nothing stopping you from adding parametric structure along with it -- perhaps in the form of random effects.

This hasn't been implemented however for classification problems, which I assume that you're doing because you're interested in SVM as a candidate.

Answer 2

Repeating from machine learning techniques for longitudinal data: the cross-validation sklearn documentation has cross-validation iterators for grouped data! See GroupKFold, LeaveOneGroupOut, and LeavePGroupsOut.

If you're interested in pure prediction, the best option is probably to use Recurrent Neural Networks. Another option is Hidden Markov Models.

Answer 3

Do you really need Random Forests, NNs, etc.for your longitudinal data? lme4 is able to handle millions of individuals:

https://cran.r-project.org/web/packages/lme4/vignettes/Theory.pdf

It can easily deal with linear mixed models, and as you can see from the link, it has also support for nonlinear mixed models (though I wouldn't expect it to be lightning quick also for the nonlinear models).