How to perform batch training using L-BFGS?

Question

I want to train a neural network for regression.

The neural network is actually composed of 4 separate child neural networks, each child neural network has a layer structure of

{input_layer: 92 nodes, 
 hidden_layer_1: 60 nodes, 
 hidden_layer_2: 60 nodes, 
 output_layer: 1 node}

As a result, the model has about $40000$ parameters to be adjusted.

I have little experience in training a neural network, so I decide not to use stochastic gradient descent method (because I learned that I have to determine hyperparameters like learning rate). The optimizer I chose is fmin_l_bfgs_b coded in scipy.optimize.

One data point in the training set takes 6.5MB dick space, so I can load at most $800$ data points as a training batch for the optimizer.

There are several problems I encounter when I build the model for regression:

1. Each time I load a batch of training examples, I need to find the max and min of each dimension of the input vector to normalize the training batch. Can I calculate the set of (max, min) over all the training examples in the first place, then use it to normalize each training batch?

2. Does it make sense to perform such 'batch L-BFGS-B' optimization for my model? For each training batch, I need to wait for about $9$ hours till convergence ($|\text{NN_output} - \text{real_value}| < \text{some_value}$). Given the number of data points I have (almost $30000$), it will take a ridiculous amount of time to train such a model.

Any helpful suggestions will be greatly appreciated!

Answer 1

1. Normalization has to be the same for all data points in every mini-batch that you use. Imagine that you have one input feature, let's say it's 3 and you calculated min=0 and max=10, and after normalization you get value 0.3 ((3 - 0) / (10 - 0)). Now, let's say in the next iteration you have input 5, but this time you calculated different min and max, let say min=2 and max=12. You will get the same normalized value as before 0.3 ((5 - 2) / (12 - 2)). You will end up having the same representation for different numbers. Even worse case, when the same value gets different signs from different normalizations. One solution will be to pre-compute min and max and re-use these values in your training. It might take awhile, but you have to do it only once.

2. L-BFGS works only in full-batch training, which means that it hasn't been designed for mini-batch training. If you cannot afford using all samples at once for training than BFGS probably not such a good choice for your problem. You can try to use simple random search or more sophisticated methods like Tree-structured Parzen Estimators (TPE). More information about hyperparameter tuning for neural networks you can find in the article Hyperparameter optimization for Neural Networks