I want to update a bias in my Neural Network using the gradient descent optimization algorithm. Unfortunately, the bias has different dimensions than the derivative of the loss function with respect to the bias. For example, the bias in the first hidden layer has dimensions 1 x hidden_size and the delta error (i.e. the derivative of the loss function with respect to the bias) has dimensions train_size x hidden_size. So I can't just subtract one from the other.

I have seen here that the author is summing the delta error over the columns, but then I don't understand why.

Could someone help me with it ?

  • 2
    $\begingroup$ I'm not sure why you are getting downvotes, this seems like a perfectly good question to me. $\endgroup$ Commented Aug 26, 2017 at 14:52
  • $\begingroup$ @MatthewDrury: I'm not sure either. Spelling things out a little more might be helpful. $\endgroup$ Commented Aug 26, 2017 at 15:07

1 Answer 1


A loss function for a statistical or machine learning model almost always averages (or sums, as the difference between a sum and an average is just a multiplicative constant) over all the training data:

$$ L(\beta) = \sum_i (y_i - \hat y_i)^2 $$

So when you take the gradient with respect to any parameter, the derivative can be pushed into the sum:

$$ \nabla L(\beta) = - 2 \sum_i (y_i - \hat y_i) \nabla \hat y_i $$

I think what you have computed is the individual gradients $\nabla \hat y_i$, so you just need to sum to account for the fact that you are minimizing the total loss across all your training data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.