9
$\begingroup$

L2 loss, together with L0 and L1 loss, are three a very common "default" loss functions used when summarising a posterior by the minimum posterior expected loss. One reason for this is perhaps that they are relatively easy to compute (at least for 1d-distributions), L0 results in the mode, L1 in the median and L2 results in the mean. When teaching, I can come up with scenarios where L0 and L1 are reasonable loss functions (and not just "default"), but I'm struggling with a scenario where L2 would be a reasonable loss function. So my question:

For pedagogical purposes, what would be an example of when L2 is a good loss function for computing a minimum posterior loss?

For L0 it is easy to come up with scenarios from betting. Say you have calculated a posterior over the total number of goals in an upcoming soccer game and you are going to make a bet where you win $$$ if you correctly guess the number of goals and lose otherwise. Then L0 is a reasonable loss function.

My L1 example is a bit contrived. You are meeting a friend who will arrive at one of many airports and then travel to you by car, the problem is that you don't know which airport (and can't call your friend because she is up in the air). Given a posterior over which airport she might land in, where is a good place to position yourself so that the distance between her and you will be small, when she arrives? Here, the point that minimizes the expected L1 loss seems reasonable, if making the simplifying assumptions that her car will travel at constant speed directly to your location. That is, a one hour wait is twice as bad as a 30 min wait.

|cite|improve this question
$\endgroup$
  • $\begingroup$ Warning: L0 does not result in the mode for continuous problems.... $\endgroup$ – Xi'an Mar 16 '15 at 15:02
  • $\begingroup$ Hmm, yeah I know that it is a bit sloppy to say L0 -> mode. $\endgroup$ – Rasmus Bååth Mar 17 '15 at 9:51
  • 2
    $\begingroup$ Looking at the inverse square law, if you have several sources of light placed so that any point in space we can choose will get negligible light from all but the nearest source, using L2 loss would be equivalent to wishing to minimise, say, the number of seconds per lumen received. I can't think of why you'd want to do that, though, rather than maximising lumens per second. $\endgroup$ – Accidental Statistician Apr 15 '15 at 16:09
4
$\begingroup$

  1. L2 is "easy." It's what you get by default if you do standard matrix methods like linear regression, SVD, etc. Until we had computers, L2 was the only game in town for a lot of problems, which is why everyone uses ANOVA, t-tests, etc. It's also easier to get an exact answer using L2 loss with many fancier methods like Gaussian processes than it is to get an exact answer using other loss functions.

  2. Relatedly, you can get the L2 loss exactly using a 2nd-order Taylor approximation, which isn't the case for most loss functions (e.g. cross-entropy, ). This makes optimization easy with 2nd-order methods like Newton's method. Lots of methods for dealing with other loss functions still use methods for L2 loss under-the-hood for the same reason (e.g. iteratively reweighted least squares, integrated nested Laplace approximations).

  3. L2 is closely related to Gaussian distributions, and the Central Limit Theorem makes Gaussian distributions common. If your data-generating process is (conditionally) Gaussian, then L2 is the most efficient estimator.

  4. L2 loss decomposes nicely, because of the law of total variance. That makes certain graphical models with latent variables especially easy to fit.

  5. L2 penalizes terrible predictions disproportionately. This can be good or bad, but it's often pretty reasonable. An hour-long wait might be four times as bad as a 30-minute wait, on average, if it causes lots of people to miss their appointments.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ Hmm, what I was after was more like a decision situation where L2 would be a reasonable loss funktion. Like the scenario similar to the two examples in my question, but for L2. $\endgroup$ – Rasmus Bååth Mar 17 '15 at 18:59
  • 1
    $\begingroup$ @RasmusBååth I'm not sure about an argument for exactly squaring the loss (apart from its connection to Gaussian data-generating processes in #3), but #5 is an argument for an accelerating loss function of some kind. To second order, any such function will match L2 loss. $\endgroup$ – David J. Harris Mar 17 '15 at 23:22
  • $\begingroup$ @DavidJ.Harris Actually, #5 is incorrect. What you would do in such a case is use L1 abs(x-y) loss to minimize frustration = time². Using (x-y)² loss for time, as you have suggested is actually going to give you a suboptimal result. $\endgroup$ – Íhor Mé Jun 7 '18 at 23:19
  • $\begingroup$ @ÍhorMé I think I must be misunderstanding you. It sounds like you're saying that the best way to minimize squared error is by minimizing the absolute loss, and not the L2 norm. $\endgroup$ – David J. Harris Jun 11 '18 at 18:46
  • $\begingroup$ @DavidJ.Harris Yeah, I was trying to make a point that this is a problem of minimizing "badness" (= time diff²) and not time spent waiting, essentially, but I think I misunderstood the thought experiment originally. Now that I'm rereading it, L2 is a legit way to move from minimizing time difference to minimizing "badness". Though, I must say, it's best for a programmer to first correctly identify what "badness" he wants to minimize, then get that value, then minimize via L1. In this case you get (time diff)² first then minimize L1 loss. Go with L2 only when you know what you're doing. $\endgroup$ – Íhor Mé Aug 27 '18 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.