I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how many samples are chosen while training. That is, SGD iteratively chooses one sample to perform forward pass followed by backpropagation to adjust the weights, as oppose to GD where the backpropagation starts only after the entire samples have been calculated in the forward pass).

My question is: When the Gradient Descent (or even mini-batch Gradient Descent) is the chosen approach, how do we represent the error from a single forward pass? Assuming that my network has only a single output neuron, is the error represented by averaging all the individual errors from each sample or by summing all of them? To me this sounds like an implementation-dependent manner but I want to know if there is such a conventional way of that.

  • $\begingroup$ Please refrain from asking the same question on both Cross Validated and Data Science SE. There is no value in dispersing answers across the two sites. $\endgroup$ – E_net4 is being targeted Dec 10 '17 at 17:44

You can use either of them, it a measure of the loss generated by the current weights of the model. If you use sums it gives you a big loss bigger your "batch", generally we use expectation to normalize the loss irrespective of the size of your "batch"

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.