# Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

276 questions
Filter by
Sorted by
Tagged with
155k views

### How to derive the ridge regression solution?

I am having some issues with the derivation of the solution for ridge regression. I know the regression solution without the regularization term is given by: $$\beta = (X^\top X)^{-1}X^\top y.$$ But ...
• 6,797
55k views

### Why does the Lasso provide Variable Selection?

I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't. Both methods minimize the residual sum of squares ...
• 1,462
9k views

### How can top principal components retain the predictive power on a dependent variable (or even lead to better predictions)?

Suppose I am running a regression $Y \sim X$. Why by selecting top $k$ principle components of $X$, does the model retain its predictive power on $Y$? I understand that from dimensionality-reduction/...
• 2,263
18k views

### The proof of shrinking coefficients using ridge regression through "spectral decomposition"

I have understood how ridge regression shrinks coefficients towards zero geometrically. Moreover I know how to prove that in the special "Orthonormal Case," but I am confused how that works in the ...
• 2,109
69k views

### Why is the L2 regularization equivalent to Gaussian prior?

I keep reading this and intuitively I can see this but how does one go from L2 regularization to saying that this is a Gaussian Prior analytically? Same goes for saying L1 is equivalent to a Laplacean ...
• 1,239
3k views

### Ridge regression for multicollinearity and outliers

I'm wondering about techniques like ridge regression with regard to both multicollinearity and outliers. My understanding is that ridge regression is primarily used for multicollinearity, but that ...
• 555
57k views

### Regularization methods for logistic regression

Regularization using methods such as Ridge, Lasso, ElasticNet is quite common for linear regression. I wanted to know the following: Are these methods applicable for logistic regression? If so, are ...
13k views

### Why does shrinkage work?

In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive ...
2k views

### Why does regularization wreck orthogonality of predictions and residuals in linear regression?

Following up on this question... In ordinary least squares, the predictions and residuals are orthogonal. $$\sum_{i=1}^n\hat{y}_i (y_i - \hat{y}_i) = 0$$ If we estimate the regression coefficients ...
• 65k
17k views

### What problem do shrinkage methods solve?

The holiday season has given me the opportunity to curl up next to the fire with The Elements of Statistical Learning. Coming from a (frequentist) econometrics perspective, I'm having trouble grasping ...
• 14.2k
109k views

### Why L1 norm for sparse models

I am reading books about linear regression. There are some sentences about the L1 and L2 norm. I know the formulas, but I don't understand why the L1 norm enforces sparsity in models. Can someone give ...
• 1,833
50k views

### Why is multicollinearity not checked in modern statistics/machine learning

In traditional statistics, while building a model, we check for multicollinearity using methods such as estimates of the variance inflation factor (VIF), but in machine learning, we instead use ...
• 781
17k views

• 8,617
7k views

### How to perform non-negative ridge regression?

How to perform non-negative ridge regression? Non-negative lasso is available in scikit-learn, but for ridge, I cannot enforce non-negativity of betas, and indeed, ...
• 641
2k views

### Is AUC via CV a good procedure for selecting optimal model?

I'm fitting a logit classifier with LASSO and cross-validation, and struggling to select the optimal model using AUC -instead of the more usual loss like binomial deviance or classification error. I ...
• 555
64k views

### What is the adjusted R-squared formula in lm in R and how should it be interpreted?

What is the exact formula used in R lm() for the Adjusted R-squared? How can I interpret it? Adjusted r-squared formulas There seem to exist several formulas to ...
9k views

### If only prediction is of interest, why use lasso over ridge?

On page 223 in An Introduction to Statistical Learning, the authors summarise the differences between ridge regression and lasso. They provide an example (Figure 6.9) of when "lasso tends to ...
• 1,202
41k views

### Ridge, lasso and elastic net

How do ridge, LASSO and elasticnet regularization methods compare? What are their respective advantages and disadvantages? Any good technical paper, or lecture notes would be appreciated as well.
• 5,222
43k views

### What is elastic net regularization, and how does it solve the drawbacks of Ridge ($L^2$) and Lasso ($L^1$)?

Is elastic net regularization always preferred to Lasso & Ridge since it seems to solve the drawbacks of these methods? What is the intuition and what is the math behind elastic net?
• 5,643
15k views

### Why do smaller weights result in simpler models in regularization?

I completed Andrew Ng's Machine Learning course around a year ago, and am now writing my High School Math exploration on the workings of Logistic Regression and techniques to optimize on performance. ...
• 511