0
$\begingroup$

Square of number < 1.0 makes that number smaller. So the gradient became smaller and learning very slow. So for me, it looks naturally to use L2 loss when the difference between target and output is more than one and use L1 when it is < 1.0. The same is true for example for Mean Relative Squared Error there we don't need to take square when a relative error is less than one. I have tested my thought on a real time series prediction dataset with different gradient descent optimizers(Adam, vanilla, Adadelta) and that gave me much faster and better convergence. What do you think. Is this a dataset-specific or should be general practice? Error surface became not smooth and not convex when using two losses, but the same is true for ReLu and that is not a problem. What are the potential dangers of this approach?

$\endgroup$
0
$\begingroup$

Changing the loss, changes the problem, so you can't objectively compare using one loss with other.

As for your idea of using a hybrid between L1 and L2 loss, Huber loss, does this, but it's the other way around, preferring L2 loss when the difference is small and L1 loss otherwise.

$\endgroup$
  • 1
    $\begingroup$ Typically we never need "squared results" - that is just a way to make learning faster as far as I know. For example, I read a research paper about time series predictions. The goal is to make accurate predictions for video views. In a research paper, people are using Relative Squared Error (mRSE). If I am combining Relative Squared Error and Relative Error I got much faster convergence and got better predictions in terms of relative predictions errors. So I don't understand why to use always l2(squared) - that is a typical choice in 90% or l1 and not combine both. $\endgroup$ – Brans Ds Jun 7 '17 at 11:48
  • $\begingroup$ @BransDs L2 error isn't "squared results"—it's Gaussian errors, which is the assumption made by linear regression. $\endgroup$ – Neil G Jun 7 '17 at 11:49
  • 1
    $\begingroup$ This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review $\endgroup$ – Antoine Jun 7 '17 at 11:57
  • 1
    $\begingroup$ @Antoine Sure it does. It may be short, but it's very simple: You cannot compare loss functions. You can compare learning algorithms with respect to a loss function. This is a common misunderstanding. Often people are told "don't start with an algorithm and then look for a problem". It's the same sort of problem with this question. $\endgroup$ – Neil G Jun 7 '17 at 11:59
  • 1
    $\begingroup$ @NeilG, (1) you are conflating the specification of a statistical model with estimation of its parameters. Maximum likelihood estimator is not the only possible one. (2) Might be poor formulation on my side. I meant comparing estimator performance where the estimators are based on different loss functions. I am not entirely sure whether that is what the OP means, though. $\endgroup$ – Richard Hardy Jun 7 '17 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.