Train an ensemble of neural networks on different datasets, what is the best way to scale the inputs?

Question

I'm training an ensemble of three neural networks using l-BFGS method for regression.

Each neural network is trained on a sub-dataset that is randomly sampled from a large dataset. Since the sub-datasets used for training are different, an input feature may have different min and max values for different models (different MinMaxScalars).

The reason that I train such an ensemble of models is I can't load the entire dataset into memory at once which is required by the l-BFGS method. So I train three different models on different samples of the dataset. By averaging the predictions of the three models, I may have a prediction as equally good as the prediction made by a model that is trained on the entire dataset.

As I mentioned above, different models may have different MinMaxScalars if I normalize the input features on the sub-dataset a model is trained on. However, there seems to be another option, I can also find the MinMaxScalar over the entire dataset, then use it to normalize input features of different models.

How should I normalize the input features? I guess using the MinMaxScalar over the entire dataset would be a better option since the models will see more of the dataset.

Answer 1

I would normalize the data using the statistics from the entire training data.

If min/max scaling is what you've determined to be the best, then I'd use the min/max from the whole training data.

I want to emphasize that you should not generate the statistics from the entire data set (i.e. including the test set), since you've then introduced data leakage and cannot trust your performance metrics on the held out test set (since it wasn't held out).