1
$\begingroup$

Summing all gradients in one mini/batch descent is a special methode or an obligation to get the correct gradient.

Does making median/mean/maxes/... methodes to have different 'global' gradient from sum is usefull ?

$\endgroup$

1 Answer 1

0
$\begingroup$

Gradients are summed because the loss functions are typically defined as the sum of the individual losses of training samples, and the differentiator distributes over the sum: $$L=\sum_{i=1}^B L(x_i,y_i)\rightarrow \nabla L=\sum_{i=1}^B \nabla L(x_i,y_i)$$

$\endgroup$
2
  • $\begingroup$ Is there like $ \frac{1}{B} * \sum {L(x_i,y_i)}$ ? $\endgroup$
    – K V
    Feb 16, 2021 at 11:34
  • $\begingroup$ The scalar multiplicand is not so important. Some definitions include it, some not. (it's not $B-1$) $\endgroup$
    – gunes
    Feb 16, 2021 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.