Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Question

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (pseudo-) independent time series of about 50 variables each, over about 100 periods. Obviously, because I'm pooling them, I'm attempting to learn a function that fits each of the few thousand time series in an average sense.

Essentially, my model is $$ \mathbf{Y}_{it} = f(\mathbf{Y}_{lags}, \Lambda) + \epsilon_{it} $$

where $\mathbf{Y}$ is a matrix of outcome variables indexed by the cross-sectional unit and time (in keras, it has dimension $i \times t \times p$ where $p$ is the number of variables), $\Lambda$ is a vector of hyperparameters -- things like regularization penalties and dropout rates, and $f$ is some time series network, like a temporal convolutional network or a LSTM.

Rather than explicitly trying to find one optimal value of $\Lambda$, I'm using the following algorithm to train $f$. It's a form of Bayesian Hyperparameter Optimization (I guess), only it allows for optimal $\Lambda$ to change over the course of training:

1. Split data into training/testing set
2. For initial $\theta$ (defining $\theta$ as the weights/parameters that make up the actual model), compute test loss, put it in $L^{old}$, and put $\theta$ in $\theta^{old}$
3. Draw from some distribution of candidate values of $\Lambda$, use to train model to get $\theta^{new}$ and $L^{new}$. If $L^{new} < L^{old}$, put $\theta^{new}$ into $\theta^{old}$. Otherwise retain $\theta^{old}$ for the next epoch.
4. Repeat (3) until satisfied

Here's my question: is there anything wrong with redefining the train/test split between epochs?

The algorithm would become:

1. Initialize $\theta$, call it $\theta^{old}$
2. Define a random train test split, then compute $L^{old}$ on the new test set
3. Draw from some distribution of candidate values of $\Lambda$, use to train model to get $\theta^{new}$ and $L^{new}$. If $L^{new} < L^{old}$, put $\theta^{new}$ into $\theta^{old}$. Otherwise retain $\theta^{old}$ for the next epoch.
4. Repeat (2) and then (3) until satisfied

Obviously I'd have to use some moving average of loss over epochs, because losses between testing sets won't be comparable.

I don't see why this should overfit necessarily. It's akin to bagging in that it uses many different train/test splits, but I'm not building many models to average. Instead I'm making the different observations "take turns" at the test set. The reason that I'm considering it is that I don't have millions of cross-sectional units, and I want to avoid bias stemming from too-big of a test set, as well as the expense of bagging.

It could be said that a given epoch could overfit to a specific observation, and then that could persist when that observation does its turn in the test set. But it seems to me that this effect would eventually wash out, especially if for each epoch I enforce that the training set and the testing set are together smaller than the total number of samples.

I'd appreciate suggestions on the algorithm! Or, has someone already invented this technique and called it something that I haven't been able to find on google? That'd be helpful too.

Edit It should be mentioned that my population has a fixed size $i$ -- new data will come in the form of more time $t$, rather than more cross-sectional units $i$. Many time series ML models only use some future period as a target, but I explicitly want my model to both explain the past (using cross-sectional variation) as well as predict the future.

To be more specific, I have about 2000 locations, from 1900 to present. The first thing I'm going to do is to train my model up to 1980, and see how well it predicts the present day. If it does, I'll follow this approach to train the model up to the present, and then project it into the future.

Answer 1

The answer is No. You should never do such a thing.

What you are proposing is this: train a model for a number of epochs until it converges. At each epoch, divide the dataset into a training and a testing set. Train on the training set and test on the test set. Repeat this process.

Before I can answer your question, let me talk about what the test set is, or rather what it is meant for. Why do you develop a machine learning system or any system for that matter? Usually, you do it for this reason: you want to develop a system that can make appropriate predictions given some input. So to do that, you go through three phases. The first phase is the development phase, where you build your system. The next is the testing phase. Here you want to check whether your system performs as desired. So you give your system some input and you ask it make predictions. You then check if the predictions made are alright. Once your system has passed your tests, it is ready for the deployment phase.

Now note that when your system has been deployed, it will get inputs that it had never seen during the training phase. This is because during training you only have access to a small subset of all real-life scenarios. Hence, you would need to know in advance how good your system performs on these "unseen" inputs. So this is where you need to be careful with the testing phase. You would want to test your model on "unseen" inputs - inputs that it has never seen during training. So your test set must consist of inputs not present in the training set. In other words, you training and test sets should have as little overlap as possible.

Now lets see what happens with your algorithm: after the first epoch, the model redivides the entire dataset into a training set and a test set. However, now the test set contains (possibly all) examples that it has already "seen" during the first epoch. Which means that your test set does not mimic deployment conditions accurately. Hence, your test accuracy on the second epoch will not be reflective of the accuracy you would get at deployment.

If you are worried about, wasting some portion of the dataset on the test set then you could use a technique called cross-validation. Here you divide your dataset in k parts. Say, k is 3. So, you would then train your model on parts 1 and 2 and test on part 3. Then, you would retrain your model (from scratch by initializing randomly) on parts 1 and 3 and test on part 2. Finally, you again retrain your model (from scratch again), but this time on parts 2 and 3 and then test it on part 1. During deployment, you run each of the three models separately and then combine their outputs in an appropriate way (e.g. weighting and linearly combining them).

So, in short you can not change the train/test split between epochs because the testing conditions must mimic the "unseen" deployment conditions which means that a model must never be trained on the testing data.