4
$\begingroup$

I have designed a simple feed-forward neural network using stochastic gradient descent. I use 22 inputs, 4 hidden layers, 1 output and am using a learning rate of 0.7 and momentum of 0.3. I have about 700 points in my training set.

As I began training, I noticed that my MSE (calculated on a validation set of about 200 points) was increasing with each epoch. I then decided to reduce the number of hidden layers I was using from 4 to 3. This - for whatever reason - worked, and the MSE dropped with each epoch and the network converged.

This leads me to lack confidence in my implementation of the backpropagation algorithms.

Why would this occur? Do extra, unneeded hidden layers prevent convergence? Could it be my algorithm?

Improve this question
$\endgroup$
4
  • $\begingroup$ How may neurons are at each hidden layer? 500 training samples is really a small number, you can easily have more parameters than samples. $\endgroup$ – jf328 Aug 18 '16 at 9:27
  • $\begingroup$ I have 22 inputs, 19 neurons in the first hidden layer, 15 in the second, 12 in the third, 8 in the fourth, and 1 output. $\endgroup$ – Bobby Aug 18 '16 at 12:55
  • 2
    $\begingroup$ Yeah, you have nearly 1000 parameters (for fully connected layers) for 500 samples. Deep networks need large sample data. Apart from what others suggest (lower learning rate etc), I guess you at least need 20k samples for your original 4 layer net $\endgroup$ – jf328 Aug 18 '16 at 14:29
  • $\begingroup$ Okay, I was wondering if I didn't have enough data... Thanks! $\endgroup$ – Bobby Aug 18 '16 at 14:37
3
$\begingroup$

Training deeper networks becomes harder as you can incur into the Vanishing Gradient Problem, especially if the activation function you are using is the sigmoid.

If you look at the derivative of the sigmoid function you will see that it's bell-shaped:sigmoid and its' derivative

This means that there is plenty of possibilities for the gradient of the sigmoid to be very low.

The gradients of the parameters of earlier layers are calculated via multiplication with the gradients of further layers. If the gradients are typically less than 1, you can see how, as you add more layers, the gradient of earlier layers gets pushed towards 0. This is explained in detail from Michael Nielsen's Neural Networks and Deep Learning book (chapter 5).

One solution would be to use alternative activation functions, such as ReLu, or Exponential Linear Unit, which do not have low gradients for large inputs.

Another "solution" would be to just employ a much larger number of epochs. Moreover, make sure that proper initialization of the weights (Xavier) is employed.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I reduced my network to three hidden layers and it took about 850 epochs to train to an MSE of 0.01. Does this seem right? $\endgroup$ – Bobby Aug 18 '16 at 14:44
  • 1
    $\begingroup$ There isn't a "typical number of epochs". I think it depends on the problem. If you have a testing MSE of 0.01 then your model might be right, I guess, but MSE depends on the magnitude of the learned quantities, so it's not really relevant. Is it a classification problem? $\endgroup$ – fstab Aug 18 '16 at 14:46
  • 1
    $\begingroup$ I mean, if you learn final quantities that are 0.001 and 0.002 then a MSE of 0.01 is huge, so it does not make any sense that you talk about MSE, unless, for example, you tell us that you are actually learning a one-hot 1-of-K category vector. $\endgroup$ – fstab Aug 18 '16 at 14:57
  • $\begingroup$ It's actually not a classification problem. It is to predict prices, so the output should be anywhere in the range of 0 to 1. $\endgroup$ – Bobby Aug 18 '16 at 15:10
  • 1
    $\begingroup$ Yes, but if the distribution of the prices is skewed towards 0, then MSE 0.01 might still be too high. If you use ELU or ReLU, then you don't need to normalize the prices from 0 to 1. $\endgroup$ – fstab Aug 18 '16 at 15:12
1
$\begingroup$

If your error function diverges try reducing your learning rate from 0.7 to say 0.007 or 0.07. More or less hidden layers should not affect convergence though the generalization power of the two would be different.

Improve this answer
$\endgroup$
2
  • $\begingroup$ When I decreased my learning rate to 0.0007 the MSE started to fall, but after about 5 epochs, it increased again. What would this mean? Also, do I have to decrease the momentum to 0.0003 as well? $\endgroup$ – Bobby Aug 17 '16 at 13:24
  • $\begingroup$ Depends totally on the gradients being propagated. If you are able to dump them, monitor them to decide on your values. Converging on the optimal set of parameters is not a menial task and I suggest spend some time on it. It is really good learning. $\endgroup$ – Amitoz Dandiana Aug 18 '16 at 7:05
1
$\begingroup$

More hidden layers shouldn't prevent convergence, although it becomes more challenging to get a learning rate that updates all layer weights efficiently. However, if you are using full-batch update, you should be able to determine a learning rate low enough to make your neural network progress or always decrease the objective function.

Assuming that you are using full-batch update, at a given iteration, in order to guarantee sufficient objective function decrease without manually specifying a learning rate, you can perform line search to find a learning rate that satisfies the two Wolfe conditions.

You can also use L-BFGS with line search to optimise your neural network efficiently. L-BFGS uses an approximation of the Hessian (second order gradient) which in a way sets a learning rate for every parameter. minFunc for Matlab and scipy.optimise for python have L-BFGS.

However, before going further I would do the following two things:

1) I would check that the implementation is correct by checking that the gradient function is correct. This link 1 shows a method for checking that the gradient function is correct. First, numerically approximate the gradient for a parameter using the function value, then compare it with the value returned by the gradient function. The two values should be approximately the same.

2) I would also compare the neural network (NN) results with easy-to-use NN code available online. The autograd python library shows a quick implementation of multi-layer perceptron and its performance on a toy example. You can set the learning rate and the number of hidden layers fairly easy.

Hope this helps!

Improve this answer
$\endgroup$
1
$\begingroup$

For a deep neural network that you mention, finding an effective local minima is the key.

As per the paper, Gülçehre, Çağlar, and Yoshua Bengio. "Knowledge matters: Importance of prior information for optimization." Journal of Machine Learning Research 17.8 (2016): 1-32.

It says

Deeper Networks are Harder Hypothesis: Although solutions may exist, effective local minima are generally more likely to hamper learning as the required depth of the architecture increases.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.