I am trying to train a deep neural network for classification, using back propagation. Specifically, I am using a convolutional neural network for image classification, using the Tensor Flow library. During training, I am experiencing some strange behaviour, and I'm just wondering whether this is typical, or whether I may be doing something wrong.

So, my convolutional neural network has 8 layers (5 convolutional, 3 fully-connected). All weights and biases are initialised at small random numbers. I then set a step size, and proceed with training with mini-batches, using Tensor Flow's Adam Optimizer.

The strange behaviour I am talking about is that for about the first 10 loops through my training data, the training loss does not, in general, decrease. The weights are being updated, but the training loss stays roughly at about the same value, sometimes going up and sometimes going down between mini-batches. It stays this way for a while, and I always get the impression that the loss is never going to decrease.

Then, all of a sudden, the training loss dramatically decreases. For example, within about 10 loops through the training data, the training accuracy goes from about 20 % to about 80 %. From then onwards, everything ends up converging nicely. The same thing happens each time I run the training pipeline from scratch, and below is a graph illustrating one example run.

enter image description here

So, what I am wondering, is whether this is normal behaviour with training deep neural networks, whereby it takes a while to "kick in". Or is it likely that there is something I am doing wrong which is causing this delay?

Thanks very much!

  • $\begingroup$ I guess I'm a little bit late to the party. But maybe I can still add some value to the conversation. Soo... to me it sounds like a sigmoid activation function. Since the derivative of the sigmoid is small for very small or very large values, the training might be slow for "saturated neurons". Unfortunately, I can't exactly tell what your CNN is like from the description you've given. $\endgroup$ Jul 16, 2017 at 21:12

1 Answer 1


The fact that the algorithm took a while to "kick-in" is not particularly surprising.

In general, the target function to be optimized behind neural networks are highly multi-modal. As such, unless you have some sort of clever set of initial values for your problem, there is no reason to believe that you will be starting on a steep descent. As such, your optimization algorithm will be almost randomly wandering until it finds a fairly steep valley to begin descending down onto. Once this has been found, you should expect most gradient-based algorithms to immediately begin narrowing into the particular mode that it is closest to.


This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .