I am here to ask something that I think it is interesting, first I just read about the shrinkage using the Ridge or Lasso regression by using the lambda as the penalty to introduce a little bias that reduces a large amount of variance. From here why lambda is doing is it flatter the slope that makes me think what if it does the opposite thing. It increases the slope by change the lambda < 0, I know we don't do that in shrinkage so my question is can it be beneficial for lambda < 0 in some expansion instead of shrinkage?. Are there any case it can be applied?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ From the Ridge Regression analysis at stats.stackexchange.com/a/220324/919, it is immediate that you must restrict $\lambda$ to exceed $-\sigma_p^2,$ the negative of the smallest singular value. For any values of $\lambda$ that are more negative than that, the objective function is indefinite, implying it can be made arbitrarily negative, which is useless. $\endgroup$– whuber ♦Commented Sep 6, 2020 at 16:20
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
6
To consider this, let's look at what the Lasso estimates of the coefficients is trying to minimize. Suppose $y_i$ is the outcome for observation $i=1,\ldots,n$ and that $x_{ki}$ is the value of covariate $k=1,\ldots,p$ for individual $i$. We are interested in estimating the vector of $p$ coefficients, $\beta=\beta_1, \ldots, \beta_p$ is a vector of $p$ coefficients, corresponding to the $p$ covariates they are coefficients for, as well as the intercept $\beta_0$. Then the lasso estimate of $\beta$ is
$\hat{\beta}^{lasso} = \underset{\beta}{\arg\min}\left\{\underset{i=1}{\overset{n}{\sum}}\left( y_i - \beta_0 - \underset{k=1}{\overset{p}{\sum}}\beta_k x_{ki}\right)^2 + \lambda \underset{k=1}{\overset{p}{\sum}} \vert\beta_k\vert \right\}$, for some $\lambda \geq 0.$.
One reason the lasso is used is due to the fact that highly correlated covariates lead to unstable estimates of their corresponding $\beta$-coefficients, when estimated through ordinary least squares (OLS). For instance, if $X_1$ and $X_2$ are highly correlated, then the OLS estimates of $\beta_1$ and $\beta_2$ will vary a lot between samples. This leads to an inflated mean squared error in the estimates. Now, in lasso regression, since $\lambda \geq 0$, we see that the coefficients are shrunk towards 0 since the penalty term "punishes" estimates that are very large. This is, in essence, why lasso can combat some of the problems of multicolinearity.
But what lappens if we force $\lambda < 0$? Well, this is equivalent to continuing to let $\lambda \geq 0$ and then minimize:
$\hat{\beta}^{lasso} = \underset{\beta}{\arg\min}\left\{\underset{i=1}{\overset{n}{\sum}}\left( y_i - \beta_0 - \underset{k=1}{\overset{p}{\sum}}\beta_k x_{ki}\right)^2 - \lambda \underset{k=1}{\overset{p}{\sum}} \vert\beta_k\vert \right\}$, for some $\lambda \geq 0.$.
(Note the minus before the penalty term, where previously there was a plus.) Now we are instead encouraging the estimated coefficients to be as large as possible. My intuition is that this would be especially true for covariates that are independent of $y_i$. So by forcing $\lambda < 0$, you would get the estimates of coefficients that are too far away from 0.
-
1$\begingroup$ @EconBoy and Phil: Note that the second optimization problem (the one with the minus) as stated does not have a minimum anymore. $\endgroup$– sebhoferCommented Oct 8, 2018 at 11:26
-
1$\begingroup$ @sebhofer Yes, you are correct. By letting an arbitrarily chosen $\beta_k$ go towards $\pm \infty$ we would get a smaller and smaller penalised sums squared. So there, as you have pointed out, would not exists a minimum. This means that unless we set constraints on how small $\lambda$ can be, the solution will always be that at least one of the coefficients "explodes" to infinity. I will edit my answer to reflect this. $\endgroup$– PhilCommented Oct 8, 2018 at 14:52
-
1$\begingroup$ On second thought, I am unsure if this is correct. Since the "OLS part" of the estimate should go towards $\infty$ as one of the coefficients (say $\beta_1$ goes towards $\pm\infty$, it is not obvious to me if the following statement is true or not: $\underset{\beta_1\rightarrow\pm\infty}{\lim}\left(\underset{i=1}{\overset{n}{\sum}}\left( y_i - \beta_0 - \underset{k=1}{\overset{p}{\sum}}\beta_k x_{ki}\right)^2 - \lambda \underset{k=1}{\overset{p}{\sum}} \vert\beta_k\vert \right) = -\infty$. $\endgroup$– PhilCommented Oct 8, 2018 at 15:03
-
1$\begingroup$ You're right, I wasn't paying attention. As you are using L1 regularisation you will get a minimum along $k$ as long as $x_{ki}$ are not 0 (otherwise you will get a kind of saddle point). In any case, it doesn't seem like a wise choice to me... $\endgroup$– sebhoferCommented Oct 8, 2018 at 15:31
-
1$\begingroup$ One reason the lasso is used is due to the fact that highly correlated covariates lead to unstable estimates of their corresponding β-coefficients, when estimated through ordinary least squares (OLS). This is incorrect, because LASSO has the same problem as OLS. Un der highly correlated covariates, LASSO estimates are unstable. Such settings are better treated by ridge regression, not LASSO. $\endgroup$ Commented Sep 6, 2020 at 13:12
$\begingroup$
$\endgroup$
Let's dumb down this concept, with no complex equations.
Let's say by some magic you have already found an optimal w, now in below equation if you want to tinker with logistic_loss or lasso, you have to either decrease one of them and subsequently increase other one. Right?
optimal w = logistic_loss + lasso
optimal w = constant if lasso(or lambda if you prefer) is increasing then logistic_loss will decrease and vice versa. Hence comes the underfitting when lasso = + infinity and underfitting when lasso = 0 and when lasso tends to -infinity, logistic loss will start to increase and hence you will overfit.
optimal w = logistic_loss + lasso
