In the beginning of the training process of a Neural Network, it's parameters, for example the weights in a Fully Connected Layer, have to initialized. There is a wide variety of schemes, how you can do that.

One of the simplest one is a constant initialization: All weights are set to the same value. The output of each neuron in the forward pass is then of course the same. But how bad is that? Given that there are other layers and regularization is used, shouldn't that mitigate this process and still allow the network to learn properly?

In this context I also often hear the phrase: "We don't do constant initialization to break symmetry" - but what that does actually mean, why do we want that?



Browse other questions tagged or ask your own question.