See this paragraph here: http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/
The instability property of the method of least absolute deviations means that, for a small horizontal adjustment of a datum, the regression line may jump a large amount. The method has continuous solutions for some data configurations; however, by moving a datum a small amount, one could “jump past” a configuration which has multiple solutions that span a region. After passing this region of solutions, the least absolute deviations line has a slope that may differ greatly from that of the previous line. In contrast, the least squares solutions is stable in that, for any small adjustment of a data point, the regression line will always move only slightly; that is, the regression parameters are continuous functions of the data.
For some reason, I can't find anything online describing this 'stability' phenomenon. Is it known under a different name?
Stability seems to refer to something like, for (x,y) dataset, "slightly nudge a single input x_i. For the L1 objective function, the slope of the prediction line changes massively, so the L1 objective is unstable."
I would really like a theory explanation for this picture included in that post: http://www.chioka.in/wp-content/uploads/2013/12/programmatic-L1-vs-L2-visualization.png