I am having a hard time understanding the Gradient Descent Rule for learning in a feedforward ANN. In particular, how do we determine the initial weight vector, and how is this weight vector adjusted after each epoch?

From what I've read, I know that we first define some error function depending on the weights, and I think that we choose the initial weight to be the minimizer of this error function. Is this right?


Typically neural network weights are initialized at random (for example: Xavier Initialization - Formula Clarification) while the biases are initialized at 0.

Gradient descent applies updates of the form $$x^{(k+1)} = x^{(k)} - \eta \nabla f(x^{(k)})$$ where ${}^{(k)}$ indicates that this is the $k$th iteration of the procedure and $\eta$ is the learning rate. Stochastic gradient descent only uses a fraction of the data to estimate $\nabla f(x^{(k)})$.

Gradient descent is an imperfect tool. Some discussion:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.