1
$\begingroup$

I have a fallowing problems: I'm training a neural network against some set of output values (regression problem). Those values are between -inf to inf and I can't normalize them, because they come continuously from a stream of data. Now I'm using MSE, but sometimes it results in very high losses and disturbs training. To avoid it I wanted to somehow limit the value of loss. Thus I came up with the fallowing loss:

$$ L(true, false) = \frac{1}{N}\sum_{i=1}^N{log((true - false)^2+1)}$$

The questions is, if there's a loss like this in the literature, because I can't find anything like that. If not, would it have properties like square loss or something different ?

$\endgroup$
8
  • $\begingroup$ Mean squared logarithmic error? Some info sklearn $\endgroup$ Dec 17, 2021 at 10:45
  • $\begingroup$ @user2974951 MSLE works only if compared values have values different than -1, if the values reach -1 then the loss function explodes $\endgroup$ Dec 17, 2021 at 10:46
  • $\begingroup$ If the issue is extreme values then maybe try some robust metrics, such as MAE or MdAE. $\endgroup$ Dec 17, 2021 at 10:53
  • $\begingroup$ @user2974951 MAE has the same issue here, giving losses about ~10^6 between true value and prediction. MdAE - can you expand the acronym ? $\endgroup$ Dec 17, 2021 at 10:55
  • $\begingroup$ M = mean, Md = median. If your loss truly varies that much from one episode to the next, then maybe your model is not well specified, very prone to overfitting? $\endgroup$ Dec 17, 2021 at 11:00

1 Answer 1

1
$\begingroup$

I am not aware of a loss function like this, but I'm looking forward to other answers giving pointers to literature.

Your other question is about the properties of your loss function. The inner part is a squared loss, which is minimized in expectation by the conditional mean of your observations. You then apply a monotonic transformation $x\mapsto \log(x+1)$ to this inner expression, so the minimizer stays the same. Overall, your loss function incentivizes your model to yield the conditional mean (unlike, say, the mean absolute error, which draws your predictions towards the conditional median). If this is the functional you want to elicit, go for it.

You may find a paper of mine (Kolassa, 2020, IJF) helpful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.