2
$\begingroup$

I have a neural network model with 20 layers. There are 30 input nodes and 5 output nodes. I am using backpropagation algorithm to train the model. I can see that in the first 5 layers, the weights are not changing. I don't understand why that is happening. Can someone point me in the direction of how to fix it?

$\endgroup$
  • $\begingroup$ How did you initialize the weights? Are your inputs continuous values? What's the typical range of these input nodes? $\endgroup$ – horaceT Oct 4 '16 at 17:25
  • $\begingroup$ @horaceT zero weights initialzation lead to symmetry. To break the symmetry, I randomly assigned values to weight. All inputs are categorical inputs. $\endgroup$ – NewBeeee Oct 4 '16 at 17:29
  • $\begingroup$ You have to be careful with weight initialization. If they're too small, it'd take many iterations to learn. If they're too large, the neurons could be saturated and would never learn. $\endgroup$ – horaceT Oct 4 '16 at 17:49
  • $\begingroup$ I don't see what's off topic about this. Questions on ANN & deep learning are on topic here. $\endgroup$ – gung - Reinstate Monica Oct 4 '16 at 18:29
  • $\begingroup$ How are the gradients in different layers changing? Usually the problem is exploding or vanishing gradients. $\endgroup$ – shark8me Mar 14 '17 at 10:00
2
$\begingroup$

There is nothing to fix. All knowledge you put into your model can be memorized in those last 15 layers. Probably your model is totally over fitting. Explanation: Back propagation works from layer to layer and tries to adjust weights to achieve expected results. If you have narrow and deep model expected result can be found just after altering 15 layers even if you have 20. It also can easily cause model over fitting that happened to you. You have high accuracy with training set, and high error with validation set. To avoid it you need to acquire much more data that will be able to "fulfill" model, or shrunk model to cause it generalize it's training.

$\endgroup$
  • $\begingroup$ Training accuracy: 0.98 and 10 Fold CV error: 0.25 $\endgroup$ – NewBeeee Oct 4 '16 at 17:17
  • $\begingroup$ I have a model with only 2 hidden layers, but still the first layer's weights don't change after training. Weird. $\endgroup$ – Jason Apr 26 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.