So I was asked a question on which central measures L1 (i.e., lasso) and L2 (i.e., ridge regression) estimated. The answer is L1=median and L2=mean. Is there any type of intuitive reasoning to this? Or does it have to be determined algebraically? If so, how do I go about doing that?
There is a simple geometric explanation for why the L1 loss function yields the median.
Recall that we are working in one dimension, so imagine a number line spreading horizontally. Plot each of the data points on the number line. Put your finger somewhere on the line; your finger will be your current candidate estimate.
Suppose you move your finger a little bit to the right, say $\delta$ units to the right. What happens to the total loss? Well, if your finger was between two data points, and you move it across a data point, you've increased the total loss by $\delta$ for each data point to the left of your finger, and decreased it by $\delta$ for each data point to the right of your finger. So, if there are more data points to the right of your finger than there are to the left, moving your finger to the right decreases the total loss. In other words, if more than half of the data points are to the right of your finger, you should move your finger to the right.
This leads to you moving your finger towards a spot where half of the data points are on the of that spot, and half are on the right. That spot is the median.
That's L1 and the median. Unfortunately, I don't have a similar, "all intuition, no algebra" explanation for L2 and the mean.
|show 1 more comment|