# How to change a weight/bias with gradient

After watching 3Blue1Brown's tutorial series, and an array of others, I'm attempting to make my own neural network from scratch.

So far, I'm able to calculate the gradient for each of the weights and biases.

Now that I have the gradient, how am I supposed to correct my weight/bias?

Should I:

After you've found the gradients, which we'll call $dW,db$ as shorthand for the gradients of the cost function with respect to the weights and biases respectively, you can do a variety of things to your weights and biases $W,b$. It depends on the specific optimization algorithm you are using. The most simple optimization algorithm is vanilla gradient descent. In this case, you apply the transformations: $$W\rightarrow W-\alpha dW$$ $$b\rightarrow b-\alpha dW$$ Where $\alpha$ is your learning rate. The learning rate governs how quickly and well your model learns. A learning weight which is too small may make your model have to take many steps to converge; however, a learning rate which is too big might make your model overshoot the optimum and never converge at all. Finding the correct learning rate is an iterative process.
• I'm not sure I understand your question, didn't I already break it down into 2 equations? In code they would look like W=W-alpha*dW and b = b-alpha*db. – enumaris May 3 '18 at 21:14