1
$\begingroup$

From my understanding the only difference between MSE and LSE is that with MSE you divide the sum of squared errors by the total number of values to get an average rather than just using the sum.

This means that when LSE is used as a loss function, the loss will scale up with the batch size.
Isn't that pretty undesirable?

Here is an example using batch size 1 vs batch size 128. Say we have a classification model with an output between 0-1 and it is really bad. The target value is 1 but it predicted a 0. Here the LSE = 1.
Now the same situation with a batch of 128. (128 predictions of 0 and 128 target values of 1) The LSE = 128! This higher batch size scenario is using the same model, so it isn't 128 times worse, but the loss is 128 times higher.

It seems like it would especially be a problem if your loss was made up of multiple different losses, each with different weighting, and one of them used LSE (cycle GAN is an example). You could find the right weighting with a given batch size, but then if you ever want to try another batch size wouldn't you have to recalculate the weighting? Whereas MSE you wouldn't?

How could this ever be desirable? Why would you want to use LSE instead of MSE when training a neural network? (LSGAN for example)

Edit: I am now way more confused after looking at several different pytorch implementations that use least squares and seeing they all use a mean, which is just MSE. Looking at the equation in the LSGAN paper I don't see anything about taking the mean, but in code implementations I do.

def LSGAN_D(real, fake):
  return (torch.mean((real - 1)**2) + torch.mean(fake**2))

def LSGAN_G(fake):
  return  torch.mean((fake - 1)**2)  

Isn't this literally just MSE?...

$\endgroup$

2 Answers 2

1
$\begingroup$

Just an extended comment, and probably not an answer:

  1. The "squared error" is a typical loss function for regression.
  2. The mean squared error (MSE) is the average of a couple of squared errors. It is frequently used as objective criterion in regression situations.
  3. The sum of squared errors is equivalent to the MSE in terms of optimization. Multiplying with a constant has no impact on the solution.
  4. I have never heard of LSE. There is an optimization technique called "ordinary least-squares" that minimizes the MSE, but it is fully unrelated to neural nets.
$\endgroup$
2
  • 1
    $\begingroup$ Least squares gan (LSGAN) was a pretty popular paper - arxiv.org/pdf/1611.04076.pdf Several other GANs adopted using least squares error. And your 3rd point is one of the reasons why I wanted to ask this question. $\endgroup$
    – Frobot
    Sep 25, 2023 at 20:57
  • $\begingroup$ Thx for the link! $\endgroup$
    – Michael M
    Sep 26, 2023 at 6:52
0
$\begingroup$

LSE is a deterministic metric, but MSE is a stochastic (statistical) metric. to choose a proper metric you should know your question. if problem is stated in statistical expression, you should use MSE, and in deterministic approaches like deterministic regression you should use LSE.

In neural network training, because output is given, deterministic approach is desirable. But in when you know statistics of output, weighting with probability metrics will give better results, which is MSE.

$\endgroup$
2
  • $\begingroup$ Both are deterministic. They are basically the exact same thing except in MSE you divide the sum of errors by a constant. $\endgroup$
    – Frobot
    Sep 25, 2023 at 22:21
  • $\begingroup$ what is constant in MSE? it is covariance which is statistical quantity. Also, you can see in wikipedia : The MSE is the second moment (about the origin) of the error. $\endgroup$ Sep 28, 2023 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.