3
$\begingroup$

If all parameters of $\mathbf{W}$ are the same, all $\partial \mathbf{W}$'s rows would also be same. But why is this bad? Can't I proceed from this?

$$ \mathbf{W}= \begin{pmatrix} a & a & a\\ a & a & a \end{pmatrix} , \frac{\partial L}{\partial \mathbf{W}}= \begin{pmatrix} b & c & d \\ b & c & d \end{pmatrix} $$

After first update,

$$ \begin{aligned} \mathbf W &= \begin{pmatrix} a -b & a-c & a-d \\ a -b & a-c & a-d \end{pmatrix} \\ \mathbf X &= \begin{pmatrix} x_1 & x_2 \end{pmatrix} \\ \mathbf {XW} &= \begin{pmatrix} x_1(a-b) + x_2(a-b) & x_1(a-c) + x_2(a-c) & x_1(a-d) + x_2(a-d) \end{pmatrix} \end{aligned} $$



(Edit) Belows are example of backpropagation of cross entropy and softmax.

enter image description here

enter image description here

Here $a_1, a_2, a_3$ are input of softmax(=output of last Affine layer)
$y_1=y_2=y_3$ but $t_1,t_2,t_3$ is all zero except one(one-hot encoding).
So $\frac{\partial L}{\partial a_1}, \frac{\partial L}{\partial a_2}, \frac{\partial L}{\partial a_3}$ is not all same and eventually it causes $\mathbf W$ not to have all same elements after first update.

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ because if each neuron has the same weights it has the same response, so it is the same as having only a single neuron. But since each neuron has the same weights it also has same gradient, so in update step the weights will stay the same. $\endgroup$
    – seanv507
    Apr 25, 2021 at 17:28
  • $\begingroup$ @seanv507 Could you explain through my example? $\endgroup$
    – firia2000
    Apr 25, 2021 at 17:35
  • $\begingroup$ Maybe you could fill out description, but I think the mistake you are making is in DL/DW: the rows are same meaning that you have a row of b (1x3) and then underneath row of c(1x3) $\endgroup$
    – seanv507
    Apr 25, 2021 at 18:13
  • $\begingroup$ so after your first update the first row of weights (input from x1 to 3 different neurons) is a-b for each column, and second row is a-c. so you end up with three neurons with the same output to any (x1,x2) input. and every subsequent update will be the same $\endgroup$
    – seanv507
    Apr 26, 2021 at 9:31
  • 1
    $\begingroup$ the assumption was you were representing the weights of hidden layer, which is the point of neural networks. If you are saying that you are representing a multinomial model (ie no hidden layer), then you are right, different inputs or different outputs mean that the weights will change. $\endgroup$
    – seanv507
    Apr 26, 2021 at 11:10

1 Answer 1

Reset to default
6
$\begingroup$

The optimization algorithms we usually use for training neural networks are deterministic. Gradient descent, the most basic algorithm, that is a base for the more complicated ones, is defined in terms of partial derivatives

$$ \theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j} J(\Theta) $$

Partial derivative tells you how does the change of the optimized function is affected by the $\theta_j$ parameter. If all the parameters are the same, they all have the same impact on the result, so will change by the same quantity. If you change all the parameters by the same value, they will keep being the same. In such a case, each neuron will be doing the same thing, they will be redundant and there would be no point in having multiple neurons. There is no point in wasting your compute repeating exactly the same operations multiple times.

When you initialize the neurons randomly, each of them will hopefully be evolving during the optimization in a different "direction", they will be learning to detect different features from the data. You can think of early layers as of doing automatic feature engineering for you, by transforming the data, that are used by the final layer of the network. If all the learned features are the same, it would be a wasted effort.

You may also be interested in reading the The Lottery Ticket Hypothesis: Training Pruned Neural Networks paper by Jonathan Frankle and Michael Carbin, who explore the hypothesis that the big neural networks are so effective because randomly initializing multiple parameters helps our luck by drawing the lucky "lottery ticket" parameters that work well for the problem.

Improve this answer
$\endgroup$
4
  • $\begingroup$ "If you change all the parameters by the same value, they will keep being the same." I think this is not true. If softmax and cross entropy are used in output layer, dL/dW becomes like above because y_i - t_i is not all same(y_i is softmax's output(all same) and t_i is 0 except answer label) $\endgroup$
    – firia2000
    Apr 26, 2021 at 10:52
  • $\begingroup$ @firia2000 I don't know what you mean by the symbols here, but it is not correct. Softmax and cross-entropy are completely deterministic functions, so if you apply them to identical values, they will return identical results. I encourage you to either do the math by pen and paper or run an example in PyTorch/TF/JAX etc and look at how to do the values do (not) change and the partial derivatives are the same. $\endgroup$
    – Tim ♦
    Apr 26, 2021 at 10:58
  • $\begingroup$ Please see edit $\endgroup$
    – firia2000
    Apr 26, 2021 at 11:18
  • $\begingroup$ @firia2000 as said in the comment by seanv507, the problem with zero-initialization applies to neural networks, where you have many neurons doing exactly the same things. It does not apply to simple models like linear, logistic, or multinomial regression, etc, in such models you can and often do initialize the parameters at zeros, but you also usually do not use gradient descent, but other optimization algorithms that are more efficient for those problems. $\endgroup$
    – Tim ♦
    Apr 26, 2021 at 12:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.