See this paragraph here: http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/

"The instability property of the method of least absolute deviations means that, for a small horizontal adjustment of a datum, the regression line may jump a large amount. The method has continuous solutions for some data configurations; however, by moving a datum a small amount, one could “jump past” a configuration which has multiple solutions that span a region. After passing this region of solutions, the least absolute deviations line has a slope that may differ greatly from that of the previous line. In contrast, the least squares solutions is stable in that, for any small adjustment of a data point, the regression line will always move only slightly; that is, the regression parameters are continuous functions of the data."

For some reason, I can't find anything online describing this 'stability' phenomenon. Is it known under a different name?

Stability seems to refer to something like, for (x,y) dataset, "slightly nudge a single input x_i. For the L1 objective function, the slope of the prediction line changes massively, so the L1 objective is unstable."

I would really like a theory explanation for this picture included in that post: http://www.chioka.in/wp-content/uploads/2013/12/programmatic-L1-vs-L2-visualization.png

  • $\begingroup$ The concept seems closely related to the breakdown point of robust statistics. $\endgroup$ – whuber Aug 21 '18 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.