46
votes
Accepted
If only prediction is of interest, why use lasso over ridge?
You are right to ask this question. In general, when a proper accuracy scoring rule is used (e.g., mean squared prediction error), ridge regression will outperform lasso. Lasso spends some of the ...
- 98.5k
45
votes
Why do we only see $L_1$ and $L_2$ regularization but not other norms?
In addition to @whuber's comments (*).
The book by Hastie et al Statistical learning with Sparsity discusses this. They also uses what is called the $L_0$ "norm" (quotation marks because this is not ...
- 82.8k
39
votes
Accepted
Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?
A natural regularization happens because of the presence of many small components in the theoretical PCA of $x$. These small components are implicitly used to fit the noise using small coefficients. ...
- 8,757
38
votes
Accepted
What are the implications of scaling the features to xgboost?
XGBoost is not sensitive to monotonic transformations of its features for the same reason that decision trees and random forests are not: the model only needs to pick "cut points" on features to split ...
- 94.1k
37
votes
L1 regression estimates median whereas L2 regression estimates mean?
This explanation is a summation of muratoa and Yves's comments on D.W.'s answer. Though it is based on calculus, I found it straightforward and easy to understand.
Assuming we have $y_1, y_2, ... y_k$...
- 471
35
votes
Accepted
Why Lasso or ElasticNet perform better than Ridge when the features are correlated
Suppose you have two highly correlated predictor variables $x,z$, and suppose both are centered and scaled (to mean zero, variance one). Then the ridge penalty on the parameter vector is $\beta_1^2 + ...
- 82.8k
34
votes
Why is glmnet ridge regression giving me a different answer than manual calculation?
The difference you are observing is due to the additional division by the number of observations, N, that GLMNET uses in their objective function and implicit standardization of Y by its sample ...
- 511
34
votes
Ridge, lasso and elastic net
To summarize, here are some salient differences between Lasso, Ridge and Elastic-net:
Lasso does a sparse selection, while Ridge does not.
When you have highly-correlated variables, Ridge regression ...
- 441
32
votes
Accepted
In regression, why not use regularization by default?
In short, regularization changes the distribution of the test statistic, rendering tests of hypothesis moot. In instances where we want to use regression to make inferences about interventions, we ...
- 39.6k
28
votes
Accepted
Why regularization parameter called as lambda in theory and alpha in python?
No difference. It's just a symbol. Sometimes mathematics uses symbols by convention, but there's no rule or requirement that you must use a certain symbol for a concept.
In this particular case, the ...
- 94.1k
26
votes
Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?
Thanks everybody for the great ongoing discussion. The crux of the matter seems to be that minimum-norm OLS is effectively performing shrinkage that is similar to the ridge regression. This seems to ...
- 107k
25
votes
Is Tikhonov regularization the same as Ridge Regression?
Carl has given a thorough answer that nicely explains the mathematical differences between Tikhonov regularization vs. ridge regression. Inspired by the historical discussion here, I thought it might ...
- 13.1k
24
votes
Is multicollinearity really a problem?
It's a problem for causal inference - or rather, it indicates difficulties in causal inference - but it's not a particular problem for prediction/forecasting (unless it's so extreme that it prevents ...
- 704
24
votes
Accepted
LASSO and ridge from the Bayesian perspective: what about the tuning parameter?
Penalized regression estimators such as LASSO and ridge are said to correspond to Bayesian estimators with certain priors.
Yes, that is correct. Whenever we have an optimisation problem involving ...
- 133k
23
votes
Accepted
The proof of equivalent formulas of ridge regression
The classic Ridge Regression (Tikhonov Regularization) is given by:
$$ \arg \min_{x} \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} + \lambda {\left\| x \right\|}_{2}^{2} $$
The claim above is that ...
- 1,157
22
votes
Why will ridge regression not shrink some coefficients to zero like lasso?
Ridge Regression
$L_{2}=(y-x\beta)^2+\lambda\sum_{i=1}^{p}\beta_{i}^2$
Will solve this equation only for one β for now and latter you can generalize this:
So, $(y-xβ)^2+λβ^2$ this is our equation ...
- 321
21
votes
Accepted
Proving Ridge Regression is strictly convex
"you can prove a function is strictly convex if the 2nd derivative is strictly greater than 0"
That's in one dimension. A multivariate twice-differentiable function is convex iff the 2nd ...
- 46.8k
20
votes
Accepted
The limit of "unit-variance" ridge regression estimator when $\lambda\to\infty$
#A geometrical interpretation
The estimator described in the question is the Lagrange multiplier equivalent of the following optimization problem:
$$\text{minimize $f(\beta)$ subject to $g(\beta) \leq ...
- 86.6k
19
votes
When to use Ridge regression and Lasso regression. What can be achieved while using these techniques rather than the linear regression model
In short, ridge regression and lasso are regression techniques optimized for prediction, rather than inference.
Normal regression gives you unbiased regression coefficients (maximum likelihood ...
- 1,188
19
votes
Why do we only see $L_1$ and $L_2$ regularization but not other norms?
I think the answer to the question depends a lot on how you define "better." If I'm interpreting right, you want to know why it is that these norms appear so frequently as compared to other options. ...
- 429
19
votes
Understanding negative ridge regression
Here is a geometric illustration of what is going on with negative ridge.
I will consider estimators of the form $$\hat{\boldsymbol\beta}_\lambda = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1}\...
- 107k
19
votes
Accepted
Why lasso for feature selection?
First, be careful in specifying what you mean by "the most important features" in a dataset. See this page for different perspectives on this issue. For example, features that are deemed "unimportant" ...
- 102k
18
votes
Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?
Here is an artificial situation where this occurs. Suppose each predictor variable is a copy of the target variable with a large amount of gaussian noise applied. The best possible model is an average ...
- 1,322
17
votes
Accepted
Why are confidence intervals and p-values not reported as default for penalized regression coefficients
Little late to the party, but in case anyone stumbles across this question in the future. . . .
Best answer: have a look at section 6 of the vignette for the penalized R package ("L1 and L2 ...
- 753
16
votes
If only prediction is of interest, why use lasso over ridge?
I think the specific setup of the example you reference is key to understanding why lasso outperforms ridge: only 2 of 45 predictors are actually relevant.
This borders on a pathological case: lasso,...
- 271
16
votes
Accepted
What would be the solution of Ridge regression, if there is an intercept?
The outline of how to derive the ridge regression solution in How to derive the ridge regression solution? is complete. I think you're simply stumbling over differences in notation; some sources ...
- 94.1k
15
votes
How to derive the ridge regression solution?
I have recently stumbled upon the same question in the context of P-Splines and as the concept is the same I want to give a more detailed answer on the derivation of the ridge estimator.
We start ...
15
votes
Regularization for ARIMA models
Answering Question 1.
Chen & Chan "Subset ARMA selection via the adaptive Lasso" (2011)* use a workaround to avoid the computationally demanding maximum likelihood estimation. Citing the paper, ...
- 69.5k
14
votes
Using regularization when doing statistical inference
There is a major difference between performing estimating using ridge type penalties and lasso-type penalties. Ridge type estimators tend to shrink all regression coefficients towards zero and are ...
- 798
13
votes
Is multicollinearity really a problem?
It's not an issue for predictive modeling when all you care about is the forecast and nothing else.
Consider this simple model:
$$y=\beta+\beta_xx+\beta_zz+\varepsilon$$
Suppose that $z=\alpha x$
...
- 62.3k
Only top scored, non community-wiki answers of a minimum length are eligible