Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite

So I was asked a question on which central measures L1 (i.e., lasso) and L2 (i.e., ridge regression) estimated. The answer is L1=median and L2=mean. Is there any type of intuitive reasoning to this? Or does it have to be determined algebraically? If so, how do I go about doing that?

share|improve this question
2  
By L1/L2 are you referring to the objective function or the constraints? If objective function then yes L1 error is minimized with the conditional median and L2 the conditional mean. If constraints (what ridge/lasso refer to) then this is the wrong way to think about this. Their "central measures" are still aiming for a conditional mean but with different penalties on $\beta$. – muratoa Aug 19 '12 at 7:10

1 Answer

up vote 4 down vote

There is a simple geometric explanation for why the L1 loss function yields the median.

Recall that we are working in one dimension, so imagine a number line spreading horizontally. Plot each of the data points on the number line. Put your finger somewhere on the line; your finger will be your current candidate estimate.

Suppose you move your finger a little bit to the right, say $\delta$ units to the right. What happens to the total loss? Well, if your finger was between two data points, and you move it across a data point, you've increased the total loss by $\delta$ for each data point to the left of your finger, and decreased it by $\delta$ for each data point to the right of your finger. So, if there are more data points to the right of your finger than there are to the left, moving your finger to the right decreases the total loss. In other words, if more than half of the data points are to the right of your finger, you should move your finger to the right.

This leads to you moving your finger towards a spot where half of the data points are on the of that spot, and half are on the right. That spot is the median.

That's L1 and the median. Unfortunately, I don't have a similar, "all intuition, no algebra" explanation for L2 and the mean.

share|improve this answer
If we are talking about a simple point estimate then it is straightforward calculus. $\frac{d}{d \beta} \frac{1}{n}\sum_{i=1}^n (y_i - \beta)^2 = -2\frac{1}{n}\sum_{i=1}^n(y_i - \beta) = 0 \Rightarrow \beta = \frac{1}{n}\sum_i y_i$ – muratoa Aug 19 '12 at 7:15
1  
@muratoa, yes, I know the calculus derivation, but the question asks specifically for an explanation that focuses on intuition and avoids algebra. I would assume that the question-asker knows the calculus derivation already, but is looking for something that provides more intuition. – D.W. Aug 19 '12 at 7:50
I thought the OP mentioned regression which suggests that he is talking about the estimate of y given x which is a conditional mean using least squares and the conditional median for mean absolute error. The same explanations should work but the problem is a little different. The calculus explanation for the mean is pretty clear and straightforward. Perhaps an explanation for the mean can be given in a similar fashion to D.W.s for the median. The sample mean is an unbiased estimate for the population mean. – Michael Chernick Aug 19 '12 at 12:07
As you move the estimate away from the sample mean the mean square error changes because of an increase in bias. The mean square error actually increases by d$^2$ when the estimate adds d to the sample mean as the candidate estimate. – Michael Chernick Aug 19 '12 at 12:08
A quick and dirty version of the algebra given by muratoa exists for the L1 case. Observe that except when $\beta = y_i$, the derivative of $| y_i -\beta |$ w.r.t $\beta$ is $-\mathrm{sgn}(y_i-\beta)$, that is $-1$ if $\beta < y_i$ and $+1$ if $\beta > y_i$. So $\frac{\mathrm{d}}{\mathrm{d}\beta} \,\frac{1}{n}\sum_i | y_i -\beta | = -\frac{1}{n}\,\sum_i \mathrm{sgn}(y_i-\beta)$, except when $\beta$ is an $y_i$. The derivative vanishes when there is the same number of positive and negative terms among the $y_i-\beta$, which roughly speaking arises when $\beta$ is the median of the $y_i$. – Yves Aug 19 '12 at 12:59
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.