Linked Questions

2
votes
1answer
3k views

Sparsity in Lasso and advantage over ridge (Statistical Learning) [duplicate]

I'm learning about the Statistical learning and in the section comparing Lasso and Ridge Regression it shows that the main difference between these two problems is the way the constraint/penalty is ...
3
votes
1answer
1k views

What is the mathematical rigorous proof that L1 regularization will give sparse solution? [duplicate]

It is given in the book Machine Learning A probabilistic Perspective, but i am not able to understand it. Can some one provide an explanation for that ? I am not clear with the way sub gradient is ...
1
vote
1answer
1k views

why l1 norm can result in variable selection but not l2 [duplicate]

In applying a penalty term with either $l_1$ or $l_2$ norm, why would the former result in variable selection but not the latter?
3
votes
1answer
1k views

Why L1 regularization can “zero out the weights” and therefore leads to sparse models? [duplicate]

I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here: Why L1 norm for sparse models [Ref. 1]. To better understand it I'm reading Google's ...
1
vote
0answers
170 views

Why does not ridge regression perform feature selection although it makes use of regularization? [duplicate]

Could somebody explain why ridge regression does not perform feature selection although it makes use of regularization? So, it penalizes the regression coefficients like LASSO does, but how come we ...
1
vote
0answers
40 views

LASSO method. Intuitively how does it select variables? [duplicate]

Intuitively how does the LASSO method select its variables? Is it based on standard econometrics?
0
votes
0answers
38 views

Comparing Ridge and Lasso Regression [duplicate]

I was thinking about main differences between ridge and lasso introducing a $\ell^2$ and $\ell^1$ penalty term respectively. The main thing is that with ridge I will keep all my features in the end ...
16
votes
2answers
10k views

Why will ridge regression not shrink some coefficients to zero like lasso?

When explaining LASSO regression, the diagram of a diamond and circle is often used. It is said that because the shape of the constraint in LASSO is a diamond, the least squares solution obtained ...
6
votes
3answers
478 views

Is there any special case where ridge regression can shrink coefficients to zero?

Are there some special cases, where the Ridge Regression can also lead to coefficients that are zero ? It is widely known, that lasso is shrinking coefficients towards or on zero, while the ridge ...
2
votes
1answer
1k views

L1 and L2 penalty vs L1 and L2 norms

I understand the usages of L1 and L2 norms however I am unsure of usage of L1 and L2 penalty when building models. From what I understand: L1: Laplace Prior L2: Gaussian Prior are two of the ...
6
votes
2answers
597 views

Is there a mathematical expression that shows how LASSO shrinks coefficients (including some to zero)?

By using singular value decomposition (SVD), I noticed from the derivation that ridge regression shrinks the coefficients by factor $\frac{D^2}{D^2+\lambda}$, where $D$ is the diagonal matrix of the ...
0
votes
2answers
735 views

Which lambda is cv.glmnet solving for?

This is my understanding of glmnet: if OLS is minimizing RSS, where $ RSS = \sum(y-\beta x)^2 $ I believe glmnet is minimizing: $ RSS - \sum(\alpha |\beta_j| + (1-\alpha) \beta_j^2) $ where $\...
2
votes
2answers
189 views

Does the order of models (and so variables) matter in nested models?

I am trying to use nested models to investigate the influence of 5 factors on my dependent variable. I am not interested in interactions, only the influence of each variable taken separately. My ...
2
votes
2answers
348 views

Binary outcome prediction with binary data

I am new to R programming and although I searched through the community, I couldn't find a similar topic, although it has to be somewhere. So a link to a similar case would be sufficient. I have a ...
0
votes
2answers
697 views

R package glmnet: Ridge selects variables

I am using the glmnet package in R. When I set the alpha value = 0, I would expect that no variables are selected. When I look at the coefficients some of them are set to zero. What could be the ...
2
votes
0answers
310 views

Implementing Lasso Regression in Numpy

I'm doing a little self study project, and am trying to implement OLS, Ridge, and Lasso regression from scratch using just Numpy, and am having problems getting this to work with Lasso regression. ...
1
vote
0answers
241 views
1
vote
1answer
85 views

Can lasso and ridge regressions theoretically have exact same solution?

Intuitively lasso leads more sparsity, but is that theoretically possible they have exact same solution vector?
0
votes
1answer
143 views

$L^1$ Regularization

Let $J(w)$ be some cost function. By adding $L^1$ regularization we get $$ \tilde{J}(w) = J(w) + \beta\sum_i|w_i| $$ To study the effect of $L^1$ regularization on the optimum weights, we can ...
3
votes
0answers
179 views

Regularization vs dropping insignificant features

Maybe I did not formulate my question properly. I would like to know how the variable selection differs between regularization and reverse-variable selection based on significance values. In short, I ...