In the PyTorch documentation for most losses, there is a parameter called reduction usually, and it is mean, but there is also a sum option. I think optimizer can handle both of the fine, so I don't understand when to use which?
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Both losses will differ by multiplication by the batch size (sum reduction will be mean reduction times the batch size). I would suggets to use the mean reduction by default, as the loss will not change if you alter the batch size. With sum reduction, you will need to ajdust hyperparameters such as learning rate of the optimizer accordingly when you change the batch size.
