Gradients are summed because the loss functions are typically defined as the sum of the individual losses of training samples, and the differentiator distributes over the sum: $$L=\sum_{i=1}^B L(x_i,y_i)\rightarrow \nabla L=\sum_{i=1}^B \nabla L(x_i,y_i)$$
• Is there like $\frac{1}{B} * \sum {L(x_i,y_i)}$ ?
• The scalar multiplicand is not so important. Some definitions include it, some not. (it's not $B-1$) Feb 16, 2021 at 11:35