6
$\begingroup$

I was having look at the orginal Tibshirani paper, JRSSB 1996. In particular, I am trying to understand his equation (6), which says that the LASSO estimates $(\hat\beta_1,\hat\beta_2)$ in the case of two regressors will be

$$\hat\beta_1=[s/2+(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2]^+\qquad\qquad(1)$$

and

$$\hat\beta_2=[s/2-(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2]^+\qquad\qquad(2)$$

where $(x)^+$ means, I think, "positive part of $x$" (i.e. $(x)^+=\max(x,0)$) and $s$ is our "budget"

$$|\hat\beta_1|+|\hat\beta_2| \leq s$$

The OLS estimates $\hat\beta^{(ols)}_1$, $\hat\beta^{(ols)}_2$ need to be positive and

$$\hat\beta^{(ols)}_1+\hat\beta^{(ols)}_2 \geq s.$$

Now, my question: if $\hat\beta_2$ is shrunk to zero, that means

$$\max[s/2-(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2,0]=0$$

or

$$s/2-(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2 < 0$$

or

$$s/2 < (\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2$$

Plugging this into (1) yields

$$\hat\beta_1 = s/2+(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2 > s/2+s/2=s$$

Hence, the "budget constraint"

$$\hat\beta_1+\hat\beta_2\leq s$$

would be violated as $\hat\beta_1>s$ and $\hat\beta_2=0$.

Where's my mistake?

$\endgroup$
1
+50
$\begingroup$

If $$\max[s/2-(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2,0]=0$$

then

$$s/2-(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2 \leq 0.$$ But you wrote $$s/2-(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2 < 0.$$ Now $$s/2-(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2 \leq 0$$ implies $$s/2\leq (\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2.$$ Next we have: $$\hat\beta_1=[s/2+(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2]^+=max[s/2+(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2,0].$$ Therefore, $$s/2+(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2\geq s/2+s/2.$$ So $$s/2+(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2\geq s \geq 0\qquad\qquad (1)$$ Hence using (1) we have: $$\hat\beta_1=max[s/2+(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2,0]=s/2+(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2 \qquad\qquad (2)$$ But you found $$\hat\beta_1>s.$$ Finally using (2) we have: $$|\hat\beta_1|+|\hat\beta_2|=|\hat\beta_1|+0=|\hat\beta_1|=|s/2+(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2| \leq s/2 +|(\hat\beta^{(ols)}_1-\hat\beta^{(ols)}_2)/2|.$$ The last in-equality can be less than or equal to $s$ and nothing is violated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.