In a basic 2 layer neural net, there are (for example) 3 inputs that each go into each neuron. If the same inputs are going into each neuron and we are applying the same optimization algorithm of gradient descent as well as the same sigmoid activation function, wouldn't they all give you the same result rendering the extra neurons useless?

What I see is that you would apply gradient descent to each of the randomly chosen weights and biases, and eventually, they would all reach the same value since all the other functions are kept constant.

Can anyone please explain what I'm missing here? Thanks!


2 Answers 2


If all weights are initialized with the same values, all neurons in each layer give you the same outputs. This is the reason that the weights are initialized with random numbers.

  • 1
    $\begingroup$ But even if the weights were initialized with random values, after gradient descent, shouldn't they theoretically converge to the same values? $\endgroup$ Sep 6, 2017 at 2:45
  • 1
    $\begingroup$ No. If you look at the equation of how weights are updated, then the first term is the initial weight from which derivative of the loss function is added or subtracted. This derivative would be the same but since the initial values of weights are different, you arrive at different values. Go through Gradient Descent here: cs231n.github.io/optimization-1 $\endgroup$
    – Jatin
    Sep 6, 2017 at 3:12
  • $\begingroup$ @Jatin Are u trying to say that derivative would be same for all weights of same layer ? I really doubt. $\endgroup$ Sep 6, 2017 at 7:57
  • $\begingroup$ Yes the derivative function is the same, so if you continuously update the function until the derivative is 0, then the value would have to be the same by that logic. If the derivative is the same then the minimum of the integral of that derivative, which is the 'initial' value must also end up being the same? $\endgroup$ Sep 7, 2017 at 0:47

Let say we have an input vector $\vec{X_0}$ and two neurons $A$ and $B$:

$F(A) = \sigma(W_{0A}*\vec X_0-\beta_A)$


$F(B) = \sigma(W_{0B}*\vec X_0-\beta_B)$

accordingly, where $\sigma$ is the activation function (let's say sigmoid) and $W$ is the weights matrix from layer 0 (input) to layer 1 (A,B).

The results of A and B would be different if the weight of A and B are different.

The gradient update won't necessary be the same:

  • First, A and B may lead in further layers to different calculations, and therefore their effect on the loss function ($\mathcal L(\vec X)$) won't be the same. For example: $\mathcal L(\vec X) = w_A*A + w_B*B -Y $

  • Moreover, even if the weight are different, the deriviation of $\frac{ \mathcal L(\vec X)}{dW}$ won't be the same for $W_{0A}$ and $W_{0B}$, and therefore the update will result in different values.

  • $\begingroup$ In your example, both W0A and W0B would be computing the same logistic regression operation with the same input vector X. Obviously, the gradient at that random W value would be different, but wouldn't the global optimum be the same and therefore both neurons perform the same function? $\endgroup$ Sep 7, 2017 at 5:30
  • $\begingroup$ No, since the outputs of A and B are effecting the final result ($\mathcal{L}$) differently. $\endgroup$
    – Dimgold
    Sep 7, 2017 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.