2
$\begingroup$

For training a neural network is there any significant difference between using the Sum Squared error or the Mean squared error as the loss function?

$\endgroup$
2
  • 1
    $\begingroup$ Algebraically, it obviously doesn't change the argmin. Sometimes that's not the only consideration, though. $\endgroup$
    – Glen_b
    Oct 9, 2021 at 4:00
  • $\begingroup$ no big difference, but regularisation hyperparameters have to be rescaled appropriately ( if searching for the hyperparameter then should find rescaled param anyway) $\endgroup$
    – seanv507
    May 28, 2023 at 0:08

1 Answer 1

0
$\begingroup$

According to the theory of optimization calculus, these are equivalent: monotonic transformations like dividing by a constant do not change the $\arg\min$ that you aim to find (that $\arg\min$ gives the network weight and bias values).

However, to do any empirical work, we need to use computers to approximate these optimizations, and the sum and mean do not behave quite the same way. For example, this answer discusses why the mean might be preferred to the sum for numerical reasons when it comes to doing the calculations on a computer like you have to do. Thus, despite the theoretical guarantees of the sum and mean having the same $\arg\min$, the numerical considerations matter for doing applied work.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.