1
$\begingroup$

Today I learned that the L1 Lasso's lambda parameter can be adjusted from small to large, with small being that Lasso basically produces the same result as least squares (no regularization/ leaves coefficients as is), and large being it shrinks all coefficients towards 0.

And so the practical approach would be having that lambda value somewhere in between, likely shrinking unimportant coeffs to 0 and keeping important ones above 0.

My question is isn't there a lambda value somewhere in that spectrum that would represent the same output a Ridge regression would output? I'm imagining this lambda value slightly smaller than shrinking any coeffs to 0, as per the definition of Ridge. If this is possible, why do we even bother with Ridge if we can just adjust the lambda value accordingly to our needs (depending on if we wanted a Lasso or Ridge output)?

$\endgroup$
4
  • $\begingroup$ Do you mean for the same $\lambda$ as the ridge regression? $\endgroup$
    – Dave
    Jan 12, 2023 at 23:57
  • $\begingroup$ No - lets say you used λ=0.5 in ridge regression and produced a set of coefficient values. My claim is that there has to exist a value of λ to input into lasso regression that will produce the same set of coefficient values as ridge just did. $\endgroup$
    – Katsu
    Jan 13, 2023 at 0:01
  • $\begingroup$ So for every $\lambda_{LASSO}$, there is $\lambda_{ridge}$ such that penalizing the LASSO regression with $\lambda_{LASSO}$ results in the same parameter estimates as penalizing the ridge regression with $\lambda_{ridge}?$ $\endgroup$
    – Dave
    Jan 13, 2023 at 0:03
  • 1
    $\begingroup$ No, this doesnt happen in general with multiple regression $\endgroup$
    – Glen_b
    Jan 13, 2023 at 1:39

1 Answer 1

3
$\begingroup$

Here is an simple illustration in R of why this is unlikely to happen when you have more than one explanatory variable and they do not have equal coefficients, taking an example where by construction $$Y = 2X_1+X_2+\epsilon$$ with $X_1,X_2,\epsilon$ iid $N(0,1)$

set.seed(2023)
nobs <- 10^5
X <- matrix(rnorm(2*nobs), nrow=nobs)
Y <- 2*X[,1] + X[,2] + rnorm(nobs)  
summary(lm(Y ~ X))
#Coefficients:
#(Intercept)           X1           X2  
#   0.003684     1.997131     0.996772  

coeffs <- matrix(nrow=201,ncol=5)
colnames(coeffs) <- c("lambda", "lasso1", "lasso2", "ridge1", "ridge2")
for (n in (-100):100){
  lasso <- glmnet(X, Y, lambda=1.1^n, family="gaussian", alpha=1) 
  ridge <- glmnet(X, Y, lambda=1.1^n, family="gaussian", alpha=0)
  coeffs[n+101, ] <- c(1.1^n, lasso$beta[,1], ridge$beta[,1])
  }   
plot( coeffs[,"lasso1"], coeffs[,"lasso2"], col="red",  type="l")
lines(coeffs[,"ridge1"], coeffs[,"ridge2"], col="blue", type="l")

Plotting the coefficients of the LASSO regression in red against each other and similarly the ridge regression coefficients in blue, your assertion would require the two lines to cross somewhere.

They do not in this example except at the extremes of zero shrinkage and total shrinkage (and would not in most other examples) because the LASSO regression model tends to reduce them by similar absolute amounts until one disappears, while the ridge regression tends to reduce both by similar proportions with both falling towards $0$ together.

enter image description here

$\endgroup$
1
  • $\begingroup$ Perfect graphical explanation to illustrate. Also clearly shows Lasso penalties are harsher than Ridge across the board. Thank you!! $\endgroup$
    – Katsu
    Jan 13, 2023 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.