2
$\begingroup$

I'm training a fully connected neural network using stochastic gradient descent (SGD). In the following graph I've ploted the training error (in the y axis) vs the epochs (in the x axis). As error function I've used the sum of squared errors $\sum_{i=0}^{N}(y - f(x_i))^2$. After running the training several times, I've found out that the resulting training error, on average, behaves very similiar to the one shown in the graph:

  1. On the beginning it's high as weights have random values,
  2. As the weights are corrected the error decreases,
  3. For a large number of epocs (in the provided graph ~1500/3500) it seems that the error reached a plateau,
  4. The correct weights are finally computed and the error reaches a value close to zero.

error vs epochs

Given these facts, I have two questions:

  1. What aspect of the network apart from the weights can cause this plateau?
  2. As I'm using SGD, shouldn't the error decrease, or in any case oscillate, on each epoch?

In the end of the training I get correct results, but it I'm curious on trying to find a way of diminishing the number of epochs this plateau lasts, so as to reduce the network training time.

$\endgroup$
0

3 Answers 3

1
$\begingroup$

This is pretty typical of saddle points. Please see

https://arxiv.org/pdf/1406.2572.pdf

for a description. We dont have a very good understanding of why loss functions do what they do; the best I can do is tell you is that the loss function you see is not a reason for alarm :)

http://lossfunctions.tumblr.com/image/129854058417

Avoiding the saddle point problem is an area of active research

$\endgroup$
1
$\begingroup$

It could be saddle points like @Sid says, especially if you work in lower dimensions. Have you tried training with another weight initialization? Also the hyperparameters of SGD may be problematic - perhaps the learning rate is too small?

Another problem (also especially in low dimensions) could be that you forgot to add a bias to your layers. This would explain a constant offset the net cannot get rid off.

If it's not this I would suspect your model's complexity is just too small to capture the training data better. Try increasing layer width or add more layers to your model and see how it behaves then.

Of course you may also simply have reached a local minimum, which is close to the global one, and will not be able to do better given your data. It's hard to say from just looking at the graph without knowing the specific application how good your error rate is.

With ANNs it is really hard to do remote diagnosis without knowing all the details as it is highly problem- and architecture-specific; even to say whether this is expected or problematic behavior is impossible for me in the end.

$\endgroup$
0
$\begingroup$

Building up on previous answers, if your issue is indeed related to a saddle point, then it might be worth trying another optimization method, as AdaGrad of Adam would perform better than SGD (see https://towardsdatascience.com/a-visual-explanation-of-gradient-descent-methods-momentum-adagrad-rmsprop-adam-f898b102325c).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.