1
$\begingroup$

I am building a multiple regression MLP (Multi-Layer Perceptron), the input is 8 weather variables collected from October-February, and the output is another weather variable.

The assumption is that the 8 variables can be used to predict the output variable. Thus far, the accuracy has been exceptional. Suspiciously exceptional... which puts doubt in my mind. In the dataset, the datapoints are organized by timestamp, beginning October and ending in February.

The potential problem is this: Is shuffling the datapoints and then splitting into training/validation/testing datasets a form of data leakage?

I've made sure to only z-scale the data based on the training data, using sklearn.preprocessing.StandardScaler()

When the data was not shuffled beforehand, the model performed less accurately on the testing and validation data. Since the data was not shuffled in this case, the testing data was only data from February. Perhaps this is because the model never "saw" typical weather conditions from February.

After training a model on the shuffled data, the Mean Squared Error for the validation and testing sets nearly halved.

$\endgroup$

1 Answer 1

3
$\begingroup$

What distinguishes time series data from other types of data is that data are collected over time (e.g. hourly, daily, weekly, monthly, etc.) and there is correlation between adjacent observations so the order of the data matters.

If you're arbitrarily shuffling time series, you're breaking the correlation structure inherent in the data, and whatever model you fit won't generalize well to your series in the future regardless of how good your error metrics look using an arbitrary split of the data. You may wish to consider validation methods designed for time series, such as time series cross-validation which evaluates your model on a rolling basis:

From 5.10 of fpp3

If you're a Python user, you may also wish to consult the documentation for sklearn or prophet).

$\endgroup$
2
  • $\begingroup$ I should clarify. The MLP sees only a single point of data at a time. Although the data is timeseries, I am not forecasting or predicting the metric on more than one data point, effectively making temporally nearby data irrelevant. $\endgroup$
    – schmibbler
    Jun 29, 2022 at 16:06
  • $\begingroup$ @schmibbler: Even if you see temporally close points as irrelevant, autocorrelation invalidates the usual assumption of independence and gives you erroneous standard errors. $\endgroup$ Jun 30, 2022 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.