I am building a multiple regression MLP (Multi-Layer Perceptron), the input is 8 weather variables collected from October-February, and the output is another weather variable.
The assumption is that the 8 variables can be used to predict the output variable. Thus far, the accuracy has been exceptional. Suspiciously exceptional... which puts doubt in my mind. In the dataset, the datapoints are organized by timestamp, beginning October and ending in February.
The potential problem is this: Is shuffling the datapoints and then splitting into training/validation/testing datasets a form of data leakage?
I've made sure to only z-scale the data based on the training data, using
When the data was not shuffled beforehand, the model performed less accurately on the testing and validation data. Since the data was not shuffled in this case, the testing data was only data from February. Perhaps this is because the model never "saw" typical weather conditions from February.
After training a model on the shuffled data, the Mean Squared Error for the validation and testing sets nearly halved.