1
$\begingroup$

There's this question in Introduction to Statistical Learning that says:

Suppose that $n=2$,$p=2$,$x_{11}=x_{12}$,$x_{21}=x_{22}$. Furthermore, suppose that $y_1+y_2=0$ and $x_{11}+x_{12}=0$ and $x_{12}+x_{22}=0$, so that the estimate for the intercept in a least squares, ridge regressions, or lasso mode is zero,$\hat{\beta_0}=0$.
(d) Argue in this setting, the lasso coefficients $\hat{\beta_1}$ and $\hat{\beta_2}$ are not unique-in other words, there are many possible solutions to the optimization problem in (c). Describe these solutions.

Information: Here (c) refers to the lasso optimization of the given data, which is:
$$(y_1-\hat{\beta_1}x_{11}-\hat{\beta_2}x_{12})^2+(y_2-\hat{\beta_1}x_{21}-\hat{\beta_2}x_{22})^2+\lambda(|\hat{\beta_1}|+|\hat{\beta_2}|)$$

Now, looking at the solution from this site: enter image description here

The Problem:
Here, the author says:

Now solutions to the original Lasso optimization problem are contours of the function $(y_1-(\hat{\beta_1}+\hat\beta_2)x_{11})^2$ that touch the Lasso-diamond $\hat\beta_1+\hat\beta_2=s$.

Now, the problem with this line is the function $(y_1-(\hat\beta_1+\hat\beta_2)x_{11})^2$. It has 4 variables, namely $\hat\beta_1$,$\hat\beta_2$,$x_{11}$ and $y_1$. So should this be plotted in 4-dimensions?
No, because in the next line the author says that :

Finally, as $\hat\beta_1$ and $\hat\beta_2$ vary along the line $\hat\beta_1+\hat\beta_2=\frac{y_1}{x_{11}}$, these contours touch the Lasso-diamond edge $\hat\beta_1+\hat\beta_2=s$ at different points.

In which the author explicitly says $\hat\beta_1+\hat\beta_2=\frac{y_1}{x_{11}}$ a line, which means there are only two dimensions. And that may also mean that the function $(y_1-(\hat\beta_1+\hat\beta_2)x_{11})^2$ might be 3-dimensional, and it's contours are represented in 2-dimensions.

But how do $x_{11}$ and $y_1$ vary in this setting, given that two dimensions are taken by $\hat\beta_1$ and $\hat\beta_2$, (as they are used to make that Lasso-diamond)?

So, basically the representation of the function $(y_1-(\hat\beta_1+\hat\beta_2)x_{11})^2$ is not clear to me.

$\endgroup$
1
$\begingroup$

The function that you say should be plotted in 4 dimensions is in fact the residual sum of squares, defined as $$ f(\hat\beta_1, \hat\beta_2, y_1, x_{11}) = (y_1 - (\hat \beta_1 + \hat\beta_2)x_{11})^2 $$ My intuitive explanation would be to treat $y$s and $x$s as parameters, i.e. fixed for any given optimization problem. Thus, both this function and the penalty term, which does not depend on the data, can be plotted in 3 dimensions (two for the $\hat\beta$s and one for the value of the function), or as contours in 2 dimensions.
(Different values of the data "parameters" would give different shapes of the RSS-contours, such as circles or ellipses instead of a line.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.