1
$\begingroup$

I'm training an ensemble of three neural networks using l-BFGS method for regression.

Each neural network is trained on a sub-dataset that is randomly sampled from a large dataset. Since the sub-datasets used for training are different, an input feature may have different min and max values for different models (different MinMaxScalars).

The reason that I train such an ensemble of models is I can't load the entire dataset into memory at once which is required by the l-BFGS method. So I train three different models on different samples of the dataset. By averaging the predictions of the three models, I may have a prediction as equally good as the prediction made by a model that is trained on the entire dataset.

As I mentioned above, different models may have different MinMaxScalars if I normalize the input features on the sub-dataset a model is trained on. However, there seems to be another option, I can also find the MinMaxScalar over the entire dataset, then use it to normalize input features of different models.

How should I normalize the input features? I guess using the MinMaxScalar over the entire dataset would be a better option since the models will see more of the dataset.

$\endgroup$
  • 1
    $\begingroup$ Why not compute the minima and maxima of the entire train set? Surely you can load a single variable at a time? On a different note, is there a reason you want to use L-BFGS? If you use some form of SGD you can simply use minibatches. $\endgroup$ – Frans Rodenburg Sep 16 at 7:31
1
$\begingroup$

I would normalize the data using the statistics from the entire training data.

If min/max scaling is what you've determined to be the best, then I'd use the min/max from the whole training data.

I want to emphasize that you should not generate the statistics from the entire data set (i.e. including the test set), since you've then introduced data leakage and cannot trust your performance metrics on the held out test set (since it wasn't held out).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.