# Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (pseudo-) independent time series of about 50 variables each, over about 100 periods. Obviously, because I'm pooling them, I'm attempting to learn a function that fits each of the few thousand time series in an average sense.

Essentially, my model is $$\mathbf{Y}_{it} = f(\mathbf{Y}_{lags}, \Lambda) + \epsilon_{it}$$

where $$\mathbf{Y}$$ is a matrix of outcome variables indexed by the cross-sectional unit and time (in keras, it has dimension $$i \times t \times p$$ where $$p$$ is the number of variables), $$\Lambda$$ is a vector of hyperparameters -- things like regularization penalties and dropout rates, and $$f$$ is some time series network, like a temporal convolutional network or a LSTM.

Rather than explicitly trying to find one optimal value of $$\Lambda$$, I'm using the following algorithm to train $$f$$. It's a form of Bayesian Hyperparameter Optimization (I guess), only it allows for optimal $$\Lambda$$ to change over the course of training:

1. Split data into training/testing set
2. For initial $$\theta$$ (defining $$\theta$$ as the weights/parameters that make up the actual model), compute test loss, put it in $$L^{old}$$, and put $$\theta$$ in $$\theta^{old}$$
3. Draw from some distribution of candidate values of $$\Lambda$$, use to train model to get $$\theta^{new}$$ and $$L^{new}$$. If $$L^{new} < L^{old}$$, put $$\theta^{new}$$ into $$\theta^{old}$$. Otherwise retain $$\theta^{old}$$ for the next epoch.
4. Repeat (3) until satisfied

Here's my question: is there anything wrong with redefining the train/test split between epochs?

The algorithm would become:

1. Initialize $$\theta$$, call it $$\theta^{old}$$
2. Define a random train test split, then compute $$L^{old}$$ on the new test set
3. Draw from some distribution of candidate values of $$\Lambda$$, use to train model to get $$\theta^{new}$$ and $$L^{new}$$. If $$L^{new} < L^{old}$$, put $$\theta^{new}$$ into $$\theta^{old}$$. Otherwise retain $$\theta^{old}$$ for the next epoch.
4. Repeat (2) and then (3) until satisfied

Obviously I'd have to use some moving average of loss over epochs, because losses between testing sets won't be comparable.

I don't see why this should overfit necessarily. It's akin to bagging in that it uses many different train/test splits, but I'm not building many models to average. Instead I'm making the different observations "take turns" at the test set. The reason that I'm considering it is that I don't have millions of cross-sectional units, and I want to avoid bias stemming from too-big of a test set, as well as the expense of bagging.

It could be said that a given epoch could overfit to a specific observation, and then that could persist when that observation does its turn in the test set. But it seems to me that this effect would eventually wash out, especially if for each epoch I enforce that the training set and the testing set are together smaller than the total number of samples.

I'd appreciate suggestions on the algorithm! Or, has someone already invented this technique and called it something that I haven't been able to find on google? That'd be helpful too.

Edit It should be mentioned that my population has a fixed size $$i$$ -- new data will come in the form of more time $$t$$, rather than more cross-sectional units $$i$$. Many time series ML models only use some future period as a target, but I explicitly want my model to both explain the past (using cross-sectional variation) as well as predict the future.

To be more specific, I have about 2000 locations, from 1900 to present. The first thing I'm going to do is to train my model up to 1980, and see how well it predicts the present day. If it does, I'll follow this approach to train the model up to the present, and then project it into the future.

The answer is No. You should never do such a thing.

What you are proposing is this: train a model for a number of epochs until it converges. At each epoch, divide the dataset into a training and a testing set. Train on the training set and test on the test set. Repeat this process.