I have a neural network model with 20 layers. There are 30 input nodes and 5 output nodes. I am using backpropagation algorithm to train the model. I can see that in the first 5 layers, the weights are not changing. I don't understand why that is happening. Can someone point me in the direction of how to fix it?

  • 1
    $\begingroup$ How did you initialize the weights? Are your inputs continuous values? What's the typical range of these input nodes? $\endgroup$
    – horaceT
    Oct 4, 2016 at 17:25
  • 1
    $\begingroup$ @horaceT zero weights initialzation lead to symmetry. To break the symmetry, I randomly assigned values to weight. All inputs are categorical inputs. $\endgroup$
    – NewBeeee
    Oct 4, 2016 at 17:29
  • $\begingroup$ You have to be careful with weight initialization. If they're too small, it'd take many iterations to learn. If they're too large, the neurons could be saturated and would never learn. $\endgroup$
    – horaceT
    Oct 4, 2016 at 17:49
  • $\begingroup$ I don't see what's off topic about this. Questions on ANN & deep learning are on topic here. $\endgroup$ Oct 4, 2016 at 18:29
  • $\begingroup$ How are the gradients in different layers changing? Usually the problem is exploding or vanishing gradients. $\endgroup$
    – shark8me
    Mar 14, 2017 at 10:00

1 Answer 1


There is nothing to fix. All knowledge you put into your model can be memorized in those last 15 layers. Probably your model is totally over fitting. Explanation: Back propagation works from layer to layer and tries to adjust weights to achieve expected results. If you have narrow and deep model expected result can be found just after altering 15 layers even if you have 20. It also can easily cause model over fitting that happened to you. You have high accuracy with training set, and high error with validation set. To avoid it you need to acquire much more data that will be able to "fulfill" model, or shrunk model to cause it generalize it's training.

  • $\begingroup$ Training accuracy: 0.98 and 10 Fold CV error: 0.25 $\endgroup$
    – NewBeeee
    Oct 4, 2016 at 17:17
  • $\begingroup$ I have a model with only 2 hidden layers, but still the first layer's weights don't change after training. Weird. $\endgroup$
    – Jason
    Apr 26, 2019 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.