# Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

607 questions
Filter by
Sorted by
Tagged with
138k views

### When should I use lasso vs ridge?

Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization ...
61k views

### Why L1 norm for sparse models

I am reading the books about linear regression. There are some sentences about the L1 and L2 norm. I know them, just don't understand why L1 norm for sparse models. Can someone use give a simple ...
40k views

### When to use regularization methods for regression?

In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS? In case this helps steer the discussion, my main interest is improving ...
22k views

### Why is ridge regression called “ridge”, why is it needed, and what happens when $\lambda$ goes to infinity?

Ridge regression coefficient estimate $\hat{\beta}^R$ are the values that minimize the $$\text{RSS} + \lambda \sum_{j=1}^p\beta_j^2.$$ My questions are: If $\lambda = 0$, then we see that the ...
5k views

### Unified view on shrinkage: what is the relation (if any) between Stein's paradox, ridge regression, and random effects in mixed models?

Consider the following three phenomena. Stein's paradox: given some data from multivariate normal distribution in $\mathbb R^n, \: n\ge 3$, sample mean is not a very good estimator of the true mean. ...
15k views

### What problem do shrinkage methods solve?

The holiday season has given me the opportunity to curl up next to the fire with The Elements of Statistical Learning. Coming from a (frequentist) econometrics perspective, I'm having trouble grasping ...
10k views

776 views

### Is there a clear set of conditions under which lasso, ridge, or elastic net solution paths are monotone?

The question What to conclude from this lasso plot (glmnet) demonstrates solution paths for the lasso estimator that are not monotonic. That is, some of the cofficients grow in absolute value before ...
5k views

### The proof of equivalent formulas of ridge regression

I have read the most popular books in statistical learning 1- The elements of statistical learning. 2- An introduction to statistical learning. Both mention that ridge regression has two formulas ...
2k views

### LASSO and ridge from the Bayesian perspective: what about the tuning parameter?

Penalized regression estimators such as LASSO and ridge are said to correspond to Bayesian estimators with certain priors. I guess (as I do not know enough about Bayesian statistics) that for a fixed ...
5k views

### Implementing ridge regression: Selecting an intelligent grid for $\lambda$?

I'm implementing Ridge Regression in a Python/C module, and I've come across this "little" problem. The idea is that I want to sample the effective degrees of freedom more or less equally spaced (like ...
9k views

### What's the typical range of possible values for the shrinkage parameter in penalized regression?

In lasso or ridge regression, one has to specify a shrinkage parameter, often called by $\lambda$ or $\alpha$. This value is often chosen via cross validation by checking a bunch of different values ...
2k views

### In regression, why not use regularization by default?

I remember reading somewhere in another post about the different viewpoints between people from statistics and from machine learning or neural networks, where one user was mentioning this idea as an ...
924 views

### Reversing ridge regression: given response matrix and regression coefficients, find suitable predictors

Consider a standard OLS regression problem$\newcommand{\Y}{\mathbf Y}\newcommand{\X}{\mathbf X}\newcommand{\B}{\boldsymbol\beta}\DeclareMathOperator*{argmin}{argmin}$: I have matrices $\Y$ and $\X$ ...