4
$\begingroup$

I get the MLE log likelihood to get a "good loss", and that DL models have non convex losses, thus leading to have local minima and so on.

However, my point is a bit different. Assume that we have a NN, and we want to do binary classification.
Independently if we use MAE, MSE, BCE, having 0 loss would mean that we are predicting always perfectly the exact label. Thus, at this point, is reasonable to assume that the loss function actually does not define "the right configuration of $\theta$, but just the "travel" that we do to get there.

Now, my question. Does the loss even care at this point? Because we have for sure that: $$ MAE_{\theta}(f(x),y) = 0 \iff MSE_{\theta}(f(x),y) = 0 \iff BCE_{\theta}(f(x),y) = 0 $$ thus the "global minima" are for sure shared between those functions.

So my question, saying "does it even care", mean something like:
ok we have shared global minima, so using one or the other, does it give us any other ice property?

In a sense like, do we have a smoother function with one loss or the other? Can we assume to have "better local minima" with one or the other? Do we get better generalization with one or the other (without other interventions like regularization/dropout etc)

(Yes I know the "BCE" helps for the saturated sigmoid section wrt MSE and MAE)

$\endgroup$
4
  • 1
    $\begingroup$ Much of your post makes it sound like you think the goal is to find where the loss is zero rather than where the loss is minimal. Certainly, zero loss sounds nice, but it is not realistic in most scenarios, and we seek out the minimum of a loss function. It is not totally clear to me what you’re asking, but I think you will get a better idea about whatever it is that is confusing you once you realize that we want to minimize loss, not eliminate loss, as loss functions need not ever equal zero. $\endgroup$
    – Dave
    Aug 28, 2022 at 15:40
  • $\begingroup$ @Dave I agree, it's been widely shown that usually generalization and loss are not the same, and having 0 loss is not the best. Also, assuming that we will converge to a global minima is also wrong from a probabilistic POV. My point is, "given that the global minima are what we would like, but probably what we will get, can we have good properties out of losses?"... for example, the smoothness, the saddle points and so on... AKA, maybe a loss is easier to optimize than others thanks to some properties of it's structure $\endgroup$
    – Alberto
    Aug 28, 2022 at 16:07
  • $\begingroup$ It would help to explain your abbreviations even though you may think everyone knows them. As far as I know, BCE is for binary $y$ whereas MAE and MSE are for continuous $y$, so I'm not sure if we're talking about the same thing. $\endgroup$ Aug 29, 2022 at 13:28
  • $\begingroup$ @ChristianHennig yes, sorry, I mean binary crossentropy, mean absolute error, mean square error... they come theoretically from the concept of maximum likelihood, but nothing stops you from using one or the other, for example for CycleGAN training, the discriminator is MAE instead of BCE, because they say "is more stable", and this is one of those properties that I'm talking about "characteristic of the loss" $\endgroup$
    – Alberto
    Aug 29, 2022 at 13:32

1 Answer 1

0
$\begingroup$

The precise form of the loss function can be important. For example, comparing mean absolute error (MAE) and mean squared error (MSE), the MSE will multiply the error value with itself, in effect giving higher weight to larger errors. This means that if you have one method that gives you very small errors for most observations but large errors for a very few, regarding MAE this will normally be better than a method that gives moderate errors for all observations, whereas regarding MSE it will normally be worse (of course this depends on the precise values).

Whether we want one or another will depend on the specific application. In some applications most if not all observations with very large errors are unreliable outliers and we don't want them to dominate the loss, whereas in other applications moderate errors can be tolerated but large errors can lead to disaster and should be avoided at all costs.

(PS: This is for continuous data not binary; for binary data the discussion is slightly different, but ultimately also there it amounts to understanding what the loss function actually implies, and how this relates to what we want in the application at hand.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.