# How to perform batch training using L-BFGS?

I want to train a neural network for regression.

The neural network is actually composed of 4 separate child neural networks, each child neural network has a layer structure of

{input_layer: 92 nodes,
hidden_layer_1: 60 nodes,
hidden_layer_2: 60 nodes,
output_layer: 1 node}

As a result, the model has about $$40000$$ parameters to be adjusted.

I have little experience in training a neural network, so I decide not to use stochastic gradient descent method (because I learned that I have to determine hyperparameters like learning rate). The optimizer I chose is fmin_l_bfgs_b coded in scipy.optimize.

One data point in the training set takes 6.5MB dick space, so I can load at most $$800$$ data points as a training batch for the optimizer.

There are several problems I encounter when I build the model for regression:

1. Each time I load a batch of training examples, I need to find the max and min of each dimension of the input vector to normalize the training batch. Can I calculate the set of (max, min) over all the training examples in the first place, then use it to normalize each training batch?

2. Does it make sense to perform such 'batch L-BFGS-B' optimization for my model? For each training batch, I need to wait for about $$9$$ hours till convergence ($$|\text{NN_output} - \text{real_value}| < \text{some_value}$$). Given the number of data points I have (almost $$30000$$), it will take a ridiculous amount of time to train such a model.

Any helpful suggestions will be greatly appreciated!

• If you reduce the number of nodes in each of the hidden layers from 60 to 30, do you obtain acceptable results? – James Phillips Dec 24 '18 at 12:21
• @James Phillips I will have a try. – meTchaikovsky Dec 24 '18 at 12:23
• What is the reason that you want to use L-BFGS in here? Yes, there's no learning rate but there are other hyperparameters. There are dedicated optimizers for NNs. – Tim Dec 24 '18 at 15:21
• My suggestion: go back to the drawing board. Ask questions like: "have I benchmarked against a simpler model?" "can I reduce my data point size?" "why should I abandon the experiences of 10's of thousands of researchers who suggest SGD for NN?" Optimizing the hyperparameters is a post-training optimization problem. You need to think about the initial optimization problem more – Cam.Davidson.Pilon Dec 24 '18 at 19:36
• @Tim I have only tried vanilla minibatch-GD for optimizing the NN. I chose three different learning rates $(0.001, 0.005, 0.01)$, but the loss was just fluctuating around. Then I tried l_bfgs_b and the loss consistently decreased (without specifying any optimization keywords), so I just naively thought maybe I can choose this for optimization... – meTchaikovsky Dec 25 '18 at 1:06