I'm training an ensemble of three neural networks using l-BFGS method for regression.
Each neural network is trained on a sub-dataset that is randomly sampled from a large dataset. Since the sub-datasets used for training are different, an input feature may have different min
and max
values for different models (different MinMaxScalars
).
The reason that I train such an ensemble of models is I can't load the entire dataset into memory at once which is required by the l-BFGS method. So I train three different models on different samples of the dataset. By averaging the predictions of the three models, I may have a prediction as equally good as the prediction made by a model that is trained on the entire dataset.
As I mentioned above, different models may have different MinMaxScalars
if I normalize the input features on the sub-dataset a model is trained on. However, there seems to be another option, I can also find the MinMaxScalar
over the entire dataset, then use it to normalize input features of different models.
How should I normalize the input features? I guess using the MinMaxScalar
over the entire dataset would be a better option since the models will see more of the dataset.