I want to train a neural network for regression.

The neural network is actually composed of 4 separate child neural networks, each child neural network has a layer structure of

{input_layer: 92 nodes, 
 hidden_layer_1: 60 nodes, 
 hidden_layer_2: 60 nodes, 
 output_layer: 1 node}

As a result, the model has about $40000$ parameters to be adjusted.

I have little experience in training a neural network, so I decide not to use stochastic gradient descent method (because I learned that I have to determine hyperparameters like learning rate). The optimizer I chose is fmin_l_bfgs_b coded in scipy.optimize.

One data point in the training set takes 6.5MB dick space, so I can load at most $800$ data points as a training batch for the optimizer.

There are several problems I encounter when I build the model for regression:

  1. Each time I load a batch of training examples, I need to find the max and min of each dimension of the input vector to normalize the training batch. Can I calculate the set of (max, min) over all the training examples in the first place, then use it to normalize each training batch?

  2. Does it make sense to perform such 'batch L-BFGS-B' optimization for my model? For each training batch, I need to wait for about $9$ hours till convergence ($|\text{NN_output} - \text{real_value}| < \text{some_value}$). Given the number of data points I have (almost $30000$), it will take a ridiculous amount of time to train such a model.

Any helpful suggestions will be greatly appreciated!

  • 1
    $\begingroup$ If you reduce the number of nodes in each of the hidden layers from 60 to 30, do you obtain acceptable results? $\endgroup$ Dec 24, 2018 at 12:21
  • $\begingroup$ @James Phillips I will have a try. $\endgroup$ Dec 24, 2018 at 12:23
  • 1
    $\begingroup$ What is the reason that you want to use L-BFGS in here? Yes, there's no learning rate but there are other hyperparameters. There are dedicated optimizers for NNs. $\endgroup$
    – Tim
    Dec 24, 2018 at 15:21
  • 2
    $\begingroup$ My suggestion: go back to the drawing board. Ask questions like: "have I benchmarked against a simpler model?" "can I reduce my data point size?" "why should I abandon the experiences of 10's of thousands of researchers who suggest SGD for NN?" Optimizing the hyperparameters is a post-training optimization problem. You need to think about the initial optimization problem more $\endgroup$ Dec 24, 2018 at 19:36
  • $\begingroup$ @Tim I have only tried vanilla minibatch-GD for optimizing the NN. I chose three different learning rates $(0.001, 0.005, 0.01)$, but the loss was just fluctuating around. Then I tried l_bfgs_b and the loss consistently decreased (without specifying any optimization keywords), so I just naively thought maybe I can choose this for optimization... $\endgroup$ Dec 25, 2018 at 1:06

1 Answer 1

  1. Normalization has to be the same for all data points in every mini-batch that you use. Imagine that you have one input feature, let's say it's 3 and you calculated min=0 and max=10, and after normalization you get value 0.3 ((3 - 0) / (10 - 0)). Now, let's say in the next iteration you have input 5, but this time you calculated different min and max, let say min=2 and max=12. You will get the same normalized value as before 0.3 ((5 - 2) / (12 - 2)). You will end up having the same representation for different numbers. Even worse case, when the same value gets different signs from different normalizations. One solution will be to pre-compute min and max and re-use these values in your training. It might take awhile, but you have to do it only once.

  2. L-BFGS works only in full-batch training, which means that it hasn't been designed for mini-batch training. If you cannot afford using all samples at once for training than BFGS probably not such a good choice for your problem. You can try to use simple random search or more sophisticated methods like Tree-structured Parzen Estimators (TPE). More information about hyperparameter tuning for neural networks you can find in the article Hyperparameter optimization for Neural Networks

  • $\begingroup$ Thank you for this helpful answer! I still have one confusion, can I train the model by L-BFGS optimizer using the largest dataset I can afford to find probably a good starting point for other optimizers? $\endgroup$ Dec 25, 2018 at 1:37
  • $\begingroup$ @meTchaikovsky It might work, but I don't think that you will gain such a large benefit from it. The incompleteness of your training dataset changes the effect of the BFGS. It's probably one of those things that easier to try and see if it works. $\endgroup$
    – itdxer
    Dec 25, 2018 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.