Train test split with time and person indexed data

Question

Setup

Consider the data in the tables below, indexed by Subject and Time $t$. Both tables show the same dataset, once ordered by Subject, once ordered by $t$ (left vs. right). The train-test split for cross validation could be done in two basic ways:

1. Split by Subject (red line in left table)
2. Split by $t$ (red line in the right table)

Problem

As time goes on, the dataset would get additional rows (e.g. for new times $t$ and new people). I'm currently trying to train a model that will predict the target (0 or 1) for those new rows. (In reality the dataset is 100000s of rows and 100s of features)

To evaluate the model I want to do cross-validation. The main question is: What are the pro's and con's of doing crossvalidation on a split along Subject (Option 1) or a split by $t$ (Option 2, "time-series crossvalidation")? Which one is preferrable? Should I even combine both since I get both new Subjects and new $t$'s over time?

Answer 1

Since this is a time-series data set the temporal issue is of utmost importance, meaning you really cannot use future data in order to predict past events. Thus, the split by Subject doesn't make much sense here since you'd be training your model with future observations but testing it on past ones (e.g. train on period {1 3 4 5 6} and test on period {2}).

The flow should follow the pattern below, with new data that comes in being ingested into the training set (blue dots) and used to predict the target variable in the future (red dots).

In light of this, I would use only the split by t, as in the right-side table.

There is a similar thread here and you can read more about time series cross validation here.

Answer 2

I would go for a train/validation/test sets if you have enough data. Choose your validation set as close as possible as the test set, which should be as close as possible to testing the performance of your model in a productive environment. In this case, is the main focus to predict the target for new subjects (then split 1 would be better) or mainly for existing subjects (split 2)? If the focus is for existing subjects (split 2) you could also add features with historical information (e.g. time elapsed since last target, etc.). Which would be more difficult to do for new customers as their history will be much shorter.

Note: if this problem is a time series analysis, you should use cross-validation only if you have no temporal issues / you de-trended all features. If you have temporal issues, you should use a train/validation/test set.

Answer 3

Analysts trained in machine learning, whose main goal is optimizing computational efficiency when processing massive data, ignore any inherent variance and structure in data by treating each observation as iid (for a recent statement of this see Sirignano, et al., Deep Learning for Mortgage Risk, https://arxiv.org/pdf/1607.02470.pdf). Sampling using the iid rule would consist of random draws from the data to create train and test groups by time in this example.

A statistician would argue that the ML method destroys structure and variance which can only be explained, preserved and recovered by partitioning train and test on, in this example, subjects.

Sirignano, et al., explicitly compare the predictive accuracy of deep learning NNs with a baseline logistic regression and conclude that, using the iid rule, logistic regression is grossly inaccurate in comparison to NNs. But is this a fair comparison? One could argue that a comparison based on iid sampling leaves LR with one hand tied behind its back. I'm not aware of papers which use the statistician's rule and does the reverse comparison. In other words, how does the predictive accuracy of the two methods (LR and NNs) compare when the inherent structure and variance in the data is preserved?

These issues bring up one of the biggest limitations of NNs: the requirement to convert multilevel categorical fields into 0,1 dummy variables. To take an extreme example residential US zip codes are an ~36,000 level massively categorical feature. Workarounds have been proposed in the literature enabling statistical, structure preserving modeling that facilitate estimating how even such a massively categorical feature explains variance in relative importance terms. On the other hand NNs would require converting this massive feature into ~36,000 dummy variables that would not only create a profusion of useless features (inevitably slowing down convergence) but also confusing the summary of results -- who cares about a specific zip code?

Here's a theoretical graphic of performance vs amount of data for which I've lost the reference:

There are some CV threads with related discussions, e.g.:

Reference showing that only deep learning algorithms benefit from using huge datasets

Fitting multilevel categorical variables with neural nets