How backpropagation through gradient descent represents the error after each forward pass

Question

I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how many samples are chosen while training. That is, SGD iteratively chooses one sample to perform forward pass followed by backpropagation to adjust the weights, as oppose to GD where the backpropagation starts only after the entire samples have been calculated in the forward pass).

My question is: When the Gradient Descent (or even mini-batch Gradient Descent) is the chosen approach, how do we represent the error from a single forward pass? Assuming that my network has only a single output neuron, is the error represented by averaging all the individual errors from each sample or by summing all of them? To me this sounds like an implementation-dependent manner but I want to know if there is such a conventional way of that.

Answer 1

You can use either of them, it a measure of the loss generated by the current weights of the model. If you use sums it gives you a big loss bigger your "batch", generally we use expectation to normalize the loss irrespective of the size of your "batch"