What do think about switching from L2-loss to L1-loss when inner difference became less then 1?

Square of number < 1.0 makes that number smaller. So the gradient became smaller and learning very slow. So for me, it looks naturally to use L2 loss when the difference between target and output is more than one and use L1 when it is < 1.0. The same is true for example for Mean Relative Squared Error there we don't need to take square when a relative error is less than one. I have tested my thought on a real time series prediction dataset with different gradient descent optimizers(Adam, vanilla, Adadelta) and that gave me much faster and better convergence. What do you think. Is this a dataset-specific or should be general practice? Error surface became not smooth and not convex when using two losses, but the same is true for ReLu and that is not a problem. What are the potential dangers of this approach?

bumped to the homepage by Community♦2 days ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

Changing the loss, changes the problem, so you can't objectively compare using one loss with other.

As for your idea of using a hybrid between L1 and L2 loss, Huber loss, does this, but it's the other way around, preferring L2 loss when the difference is small and L1 loss otherwise.

• Typically we never need "squared results" - that is just a way to make learning faster as far as I know. For example, I read a research paper about time series predictions. The goal is to make accurate predictions for video views. In a research paper, people are using Relative Squared Error (mRSE). If I am combining Relative Squared Error and Relative Error I got much faster convergence and got better predictions in terms of relative predictions errors. So I don't understand why to use always l2(squared) - that is a typical choice in 90% or l1 and not combine both. – Brans Ds Jun 7 '17 at 11:48
• @BransDs L2 error isn't "squared results"—it's Gaussian errors, which is the assumption made by linear regression. – Neil G Jun 7 '17 at 11:49
• This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review – Antoine Jun 7 '17 at 11:57
• @Antoine Sure it does. It may be short, but it's very simple: You cannot compare loss functions. You can compare learning algorithms with respect to a loss function. This is a common misunderstanding. Often people are told "don't start with an algorithm and then look for a problem". It's the same sort of problem with this question. – Neil G Jun 7 '17 at 11:59
• @NeilG, (1) you are conflating the specification of a statistical model with estimation of its parameters. Maximum likelihood estimator is not the only possible one. (2) Might be poor formulation on my side. I meant comparing estimator performance where the estimators are based on different loss functions. I am not entirely sure whether that is what the OP means, though. – Richard Hardy Jun 7 '17 at 13:57

There are a couple of problems with your plan. While your goal of improving speed is laudable is it appropriate for the problem at hand.

The first problem is that if you are minimizing $$f(\theta,\hat{\theta})$$ and you are trying to substitute solving it as $$g(\theta,\hat{\theta})$$ then the question would become "what about this substitution would make the result acceptable to whoever needs the result?"

The second problem, and this gets brushed aside way too much, is that the loss function is the loss to be experienced if you get a bad sample. Why are you using the loss function that you are using? If your answer happens to be "well everyone uses it" or, "it's the standard default" then you may be solving the wrong problem anyway. What would you lose if you got a bad sample.

Let me provide you an example. I believe this either comes from Lindley or Jeffreys originally.

Let us imagine you had to lay a cable as a single piece. For whatever reason, you cannot splice two together. Until recently, that was the case for fiber optic cable and is often the case with concrete. The true value is $$\theta=1000.$$ You will use an estimator to determine how much cable to purchase and deliver.

The estimation tool has a known standard deviation of one unit for this type of problem. You decide to use quadratic loss because you know the data is normally distributed and the estimator will be unbiased.

The cost of cable is $1000 per unit of length. There is no recovery income if too little is purchased. It cannot be resold. If you buy too little, then you must simply discard it and buy more and try again. If your unbiased estimator produces a result of 999, then your loss is \$999,000. You must then purchase another \$1,000,000 worth of cable. If your unbiased estimator produces a result of 1001, then you cut off the last unit of length and use the remaining 1000 in the actual problem. Your loss is$1000.

An unbiased estimator will produce a catastrophic loss fifty percent of the time. Your expected loss would be around 498 thousand dollars (rounded). Biased estimators fall off to a meaninglessly close to zero expected loss to the right depending on your sampling assumptions.

If time is of the essence, for example, if in robotic surgery time delays could cause death, then you could build a complexity penalty or a sample size penalty in to account for the role time plays. Nonetheless, your solution is unlikely to be optimal in any sense if that were the case.

You should be asking the question, "what do I lose if I am wrong and how do I know that?" That is your loss function.

Loss functions only appear to be cardinal measures. A loss function is a negative utility function. As a utility function, comparisons are on ordinal rankings and not cardinal. You cannot compare a quadratic loss to an absolute linear loss or any other except by stating "loss function $$f$$ is better for me than loss function $$g$$ because..."

If $$f\succ{g}$$ then $$f$$ is better than $$g$$ even if the total loss, in a particular instance, for $$g$$ is 50 units and the total loss for $$f$$ is 100 units$$^2$$.

• @Brans Ds : Take the the estimation of density parameter case with asymmetric density. Squared loss, results in a mean and absolute loss ( the absolute value ), results in the median. You have an interesting idea but what its results are, are not clear. As the other answers and comments mentioned, what is it getting you ? Based on your question, it sounds like you want to reach zero faster but what you're really trying to do is minimize a loss function rather than get to zero. They're not the same thing. – mlofton yesterday