0
$\begingroup$

i am a beginner in this from ai and i am learning about gradient descent and its update rule

Is it true that the same gradient is applied to each weight of the network at each step? that is, for example, the gradient is 0.5 because w1 has an error of 0.5

then for weight 1 is
w1 = w1-0.5
and for weight 2 it is
w2 = w2-0.5

If each weight is updated by applying the same gradient to all the weights of the network, how does it avoid entering a loop? that is to say w1 has an error of 0.5 and w2 has an error of 0 then now the weights are updated

w1 = w1-0.5
w2 = w2-0.5

now w1 has an error of 0 and w2 has an error of -0.5 then now the weights are updated

w1 = w1 - (- 0.5)
w2 = w2 - (- 0.5)

now w1 has an error of 0.5 and w2 has an error of 0 and so it enters a loop, how can this be avoided?

or each weight has a different gradient? ex:

w1 = w1 -  0.5
w2 = w2 -  0.2
$\endgroup$
0
$\begingroup$

Your post contains a misconception -- the gradient vector has an entry for each weight. These entries will usually be different, but it's not impossible for 2 weights to have the same gradient.

Your concern about "entering a loop" is a valid one. It most often arises when the weight are all initialized to the same value (e.g. all weights start at 0). The fix is easy enough: initialize weights randomly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.