1
$\begingroup$

In a basic 2 layer neural net, there are (for example) 3 inputs that each go into each neuron. If the same inputs are going into each neuron and we are applying the same optimization algorithm of gradient descent as well as the same sigmoid activation function, wouldn't they all give you the same result rendering the extra neurons useless?

What I see is that you would apply gradient descent to each of the randomly chosen weights and biases, and eventually, they would all reach the same value since all the other functions are kept constant.

Can anyone please explain what I'm missing here? Thanks!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

If all weights are initialized with the same values, all neurons in each layer give you the same outputs. This is the reason that the weights are initialized with random numbers.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ But even if the weights were initialized with random values, after gradient descent, shouldn't they theoretically converge to the same values? $\endgroup$
    – Rithwik Sudharsan
    Commented Sep 6, 2017 at 2:45
  • 1
    $\begingroup$ No. If you look at the equation of how weights are updated, then the first term is the initial weight from which derivative of the loss function is added or subtracted. This derivative would be the same but since the initial values of weights are different, you arrive at different values. Go through Gradient Descent here: cs231n.github.io/optimization-1 $\endgroup$
    – Jatin
    Commented Sep 6, 2017 at 3:12
  • $\begingroup$ @Jatin Are u trying to say that derivative would be same for all weights of same layer ? I really doubt. $\endgroup$
    – pseudo_teetotaler
    Commented Sep 6, 2017 at 7:57
  • $\begingroup$ Yes the derivative function is the same, so if you continuously update the function until the derivative is 0, then the value would have to be the same by that logic. If the derivative is the same then the minimum of the integral of that derivative, which is the 'initial' value must also end up being the same? $\endgroup$
    – Rithwik Sudharsan
    Commented Sep 7, 2017 at 0:47
0
$\begingroup$

Let say we have an input vector $\vec{X_0}$ and two neurons $A$ and $B$:

$F(A) = \sigma(W_{0A}*\vec X_0-\beta_A)$

and

$F(B) = \sigma(W_{0B}*\vec X_0-\beta_B)$

accordingly, where $\sigma$ is the activation function (let's say sigmoid) and $W$ is the weights matrix from layer 0 (input) to layer 1 (A,B).

The results of A and B would be different if the weight of A and B are different.

The gradient update won't necessary be the same:

  • First, A and B may lead in further layers to different calculations, and therefore their effect on the loss function ($\mathcal L(\vec X)$) won't be the same. For example: $\mathcal L(\vec X) = w_A*A + w_B*B -Y $

  • Moreover, even if the weight are different, the deriviation of $\frac{ \mathcal L(\vec X)}{dW}$ won't be the same for $W_{0A}$ and $W_{0B}$, and therefore the update will result in different values.

Improve this answer
$\endgroup$
2
  • $\begingroup$ In your example, both W0A and W0B would be computing the same logistic regression operation with the same input vector X. Obviously, the gradient at that random W value would be different, but wouldn't the global optimum be the same and therefore both neurons perform the same function? $\endgroup$
    – Rithwik Sudharsan
    Commented Sep 7, 2017 at 5:30
  • $\begingroup$ No, since the outputs of A and B are effecting the final result ($\mathcal{L}$) differently. $\endgroup$
    – Dimgold
    Commented Sep 7, 2017 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.