How to perform batch training using L-BFGS? I want to train a neural network for regression. 
The neural network is actually composed of 4 separate child neural networks, each child neural network has a layer structure of 
{input_layer: 92 nodes, 
 hidden_layer_1: 60 nodes, 
 hidden_layer_2: 60 nodes, 
 output_layer: 1 node}

As a result, the model has about $40000$ parameters to be adjusted. 
I have little experience in training a neural network, so I decide not to use stochastic gradient descent method (because I learned that I have to determine hyperparameters like learning rate). The optimizer I chose is fmin_l_bfgs_b coded in scipy.optimize.
One data point in the training set takes 6.5MB dick space, so I can load at most $800$ data points as a training batch for the optimizer.
There are several problems I encounter when I build the model for regression:


*

*Each time I load a batch of training examples, I need to find the max and min of each dimension of the input vector to normalize the training batch. Can I calculate the set of (max, min) over all the training examples in the first place, then use it to normalize each training batch? 

*Does it make sense to perform such 'batch L-BFGS-B' optimization for my model? For each training batch, I need to wait for about $9$ hours till convergence ($|\text{NN_output} - \text{real_value}| < \text{some_value}$). Given the number of data points I have (almost $30000$), it will take a ridiculous amount of time to train such a model.
Any helpful suggestions will be greatly appreciated! 
 A: *

*Normalization has to be the same for all data points in every mini-batch that you use. Imagine that you have one input feature, let's say it's 3 and you calculated min=0 and max=10, and after normalization you get value 0.3 ((3 - 0) / (10 - 0)). Now, let's say in the next iteration you have input 5, but this time you calculated different min and max, let say min=2 and max=12. You will get the same normalized value as before 0.3 ((5 - 2) / (12 - 2)). You will end up having the same representation for different numbers. Even worse case, when the same value gets different signs from different normalizations. One solution will be to pre-compute min and max and re-use these values in your training. It might take awhile, but you have to do it only once.

*L-BFGS works only in full-batch training, which means that it hasn't been designed for mini-batch training. If you cannot afford using all samples at once for training than BFGS probably not such a good choice for your problem. You can try to use simple random search or more sophisticated methods like Tree-structured Parzen Estimators (TPE). More information about hyperparameter tuning for neural networks you can find in the article Hyperparameter optimization for Neural Networks
