Skip to main content

Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

207 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9 votes
0 answers
754 views

The Regularization Path for Smoothing Splines

I've got a potentially interesting question. Does anyone know if R already has a package for calculating the entire regularization path of the smoothing spline? That is, for: $$\hat{f}_{\lambda}=...
Nick Thieme's user avatar
  • 1,274
7 votes
0 answers
322 views

Nonnegative identity-link Poisson regression with ridge or fused ridge penalty

I would like to fit nonnegative identity-link Poisson regression models with a ridge or fused ridge penalty, i.e. with nonnegativity constraints on the fitted coefficients, Poisson error noise & a ...
Tom Wenseleers's user avatar
7 votes
0 answers
1k views

Geometrical interpretation of L1 regression

I have found the following image (or a similar version) in a lot of books related to penalized linear models. I get the insight of this image. The ellipsoids are the solution of the linear regression ...
Álvaro Méndez Civieta's user avatar
7 votes
0 answers
291 views

Reference Request: Information Geometry for Ridge Regression

I am reading the book "regression estimators" by Gruber 2010 where he uses this technique to compare Ridge Regressors, however he concentrates on deriving the mathematical results without ...
Baz's user avatar
  • 1,763
6 votes
0 answers
100 views

Generalize the 1SE rule to elastic net

When you do LASSO or ridge regression, and pick the hyperparameter using cross-validation, the 1SE rule suggest to select not the best CV result but the one with the most penalization that's still ...
Björn's user avatar
  • 33k
6 votes
0 answers
248 views

Why does fitting the hyperparameter of Ridge regression at the same time as the model parameters does not lead to a vanishing hyperparameter?

I have been simulating some quadratic data with some noise (constant for all points) into it. I am fitting those data with a polynomial fit with Ridge regression. To find the best hyperparameter, I ...
Shamaz's user avatar
  • 161
5 votes
0 answers
883 views

How to consider interaction terms in the ridge / lasso / elastic net?

I would like to ask a question about how to consider interaction terms in my penalized regression? My primary goal is to build the model to predict. I think in the conventional GLM, we run the model ...
Qiang Super's user avatar
5 votes
0 answers
835 views

Results of cv.glmnet in R versus RidgeCV in scikit-learn

I'm having trouble reconciling different values for the ridge parameter that minimizes mean squared error when using RidgeCV in scikit-learn (Python) and cv.glmnet (R). First a few things to note: ...
optimusLime's user avatar
5 votes
0 answers
1k views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
Dima Ku's user avatar
  • 341
5 votes
0 answers
2k views

Why would concatenating feature vectors lead to better estimates?

I wish to estimate the state of a system from two separate and disparate observations. A simple approach that I have seen in some literature is to combine the feature vectors (observations) by simply ...
Josh's user avatar
  • 605
4 votes
1 answer
257 views

Should interactions also be scaled in LASSO/Ridge, or just constituent covariates?

I understand that in LASSO/Ridge it is best practice to scale covariates so that no single covariate dominates the penalized norm. However, when entering interaction terms, it is unclear whether only ...
John's user avatar
  • 165
4 votes
0 answers
75 views

Proof of invariant angle between $Y$ and $\hat Y$ in $L^2$ regularisation

On this site is the following question which claims that the $L^2$ regularised OLS preserves the angle between $\hat Y$ and $Y$ irrespective of the value $\lambda$. I have not found any source that ...
Ice Tea's user avatar
  • 345
4 votes
0 answers
152 views

Weird glmnet ridge regression results with an uncentered design matrix

I was recently trying to figure out what glmnet's ridge regression is doing (7,000 lines of Fortran are no fun) and am confused by its behavior with an uncentered design matrix $X$. I am aware that ...
const-ae's user avatar
  • 291
4 votes
0 answers
326 views

How to prove the equivalence between constrained form and Lagrange form for lasso and ridge regression?

How to prove the equivalence between constrained form and Lagrange form for lasso and ridge regression? Given lasso (constrained form): $$\underset{\beta}{\min}{(\frac{1}{2N}||y-x\beta||_2^2)} \...
FantasticAI's user avatar
4 votes
0 answers
604 views

Benchmark priors for Bayesian ridge regression

Consider a Bayesian linear regression model $$\mathbf{Y=X\beta} + \boldsymbol{\varepsilon}$$ where $\mathbf{Y} \in \mathbb{R}^n$ and $\mathbf{X} \in \mathbb{R}^{n,p}$ are given, $\boldsymbol{\...
PAM's user avatar
  • 311
4 votes
0 answers
912 views

MSE of an individual coefficient from ridge or lasso vs. OLS

Consider a multiple regression model $$ y = X\beta + \varepsilon $$ with $K$ regressors in $X$. If the model is correctly specified, the OLS estimator $\hat\beta_{OLS}=(X'X)^{-1}X'y$ will be the ...
Richard Hardy's user avatar
4 votes
0 answers
226 views

Regularisation parameter Ridge Regression

Background: I am trying to apply ridge regression in on-line mode. In batch learning we use cross-validation to calculate regularisation parameter. Is there a way to calculate regularisation ...
Waqas's user avatar
  • 426
4 votes
0 answers
1k views

Understanding Kernel Ridge Regression and How It Works (and Implementing it in R)

I am trying to understand how KRR works for drug-protein-interaction and many aspects of it seem very confusing. Supposing I have a data set as follows of Drug-Protein interactions; values show how ...
l..'s user avatar
  • 297
4 votes
0 answers
357 views

Different behaviors for different Ridge implementations in R

I am having trouble reconciling the different behavior of different Ridge implementations in R. As the following code demonstrates it seems that MASS:lm.ridge ...
JohnRos's user avatar
  • 5,704
3 votes
0 answers
29 views

Effective degrees of freedom for residual variance in ridge regression

The definition of the effective degrees of freedom (dof) in Ridge Regression via the trace of the "hat matrix" is well known (see e.g. Hastie and Tibshirani's Generalized Additive Models). ...
Markus Loecher's user avatar
3 votes
0 answers
27 views

For variable selection, would a viable alternative to using lasso be to use ridge with a threshold, or is switching to elastic net preferred?

A similar question was asked here Why can't ridge regression provide better interpretability than LASSO?, and the answer suggested that a main difference between lasso and ridge is that a zero ...
another_student's user avatar
3 votes
0 answers
85 views

Geometric intuition for how ridge ($L_2$) regularization helps under multicollinearity

We have some nice posts (1, 2 and likely more) illustrating multicollinearity geometrically. Now, ridge regression ($L_2$ regularization) is known to be a remedy of multicollinearity. What is the ...
Richard Hardy's user avatar
3 votes
0 answers
154 views

Equivalence between Gaussian Process Regression and Kernel Ridge Regression

Consider the model $$ y(\mathbf{X}) = f(\mathbf{X}) + \epsilon, $$ where $\mathbf{X}$ is a given $n\times D$ matrix, and where $\epsilon \sim \mathcal{N}(0, \sigma^{2}I_{N})$ is iid Gaussian and is ...
Akusa's user avatar
  • 331
3 votes
0 answers
338 views

Ridge Regression Alpha/Lambda: Basic Characteristics?

I fear this is an ill-posed question that has been asked a million times, but what are the basic characteristics of the penalty multiplier (usually called $\lambda$ or $\alpha$) in Ridge Regression (...
rubikscube09's user avatar
3 votes
0 answers
2k views

Why would one want to choose lambda.1se for ridge regression in glmnet?

In R, choosing lambda.1se over lambda.min to get a more parsimonious model is common. This post (and this) also indicated that ...
Blain Waan's user avatar
  • 3,595
3 votes
0 answers
351 views

A theoretical explanation why ridge is superior to lasso in non-sparse models

There are several posts about the comparison of lasso vs. ridge. However I didn't find an explanation to my question. My question is why ridge is generating lower prediction errors in cases where the ...
Leo96's user avatar
  • 505
3 votes
0 answers
1k views

Relationship between regularization parameter in Ridge/Lasso with budget constraint

The equation for lasso and ridge regression are given as follows in the ISLR textbook: The dual form of the above equations are given in terms of budget as below: I am wondering if there is a ...
bespectacled's user avatar
3 votes
0 answers
585 views

Standardization in penalized regression using glmnet

I want to run a penalized multinomial logit and logit regression using the glmnet package in R. I understand, that before fitting the penalized model, one should ...
Jogi's user avatar
  • 646
3 votes
0 answers
212 views

Bayesian "confidence intervals" for non-spline ridge regression?

Wahba (1983) and Silverman (1985) show that the quadratic penalty term on a smoothing spline is akin to a bayesian prior on the smoothness of the model. Nychka (1988) is another key reference. This ...
generic_user's user avatar
  • 13.5k
3 votes
0 answers
317 views

How can $X^TX$ be decomposed?

Nikolaenko et al. claims that in ridge regression $A\beta=b$, where $A=X^TX+\lambda I \in R^{d\times d}$ and $b=X^Ty \in R^d$ (page 3), it can be decomposed into: $$A=\sum\limits_{i=1}^{n}A_i+\...
xtt's user avatar
  • 744
3 votes
0 answers
32 views

Is it needed to regularize in case you know your data is generated by a model of your model class?

Assume we have a dataset $X_{full}$ with labels $y_{full}$. We train a kernel ridge regression model on this data with the Gaussian kernel. This model is used to generate predictions on the whole ...
Tom's user avatar
  • 1,062
3 votes
0 answers
333 views

Meaning and significance testing of coefficients in lasso/ridge regression

Can somebody explain the importance of significance testing in ridge/lasso regression? Is it necessary to do it? And, how can we interpret the coefficients of ridge regression which are penalized?
shruti's user avatar
  • 31
3 votes
0 answers
788 views

Finding standard error of beta coefficients in ridge regression using lambda

I need to get the standard errors of coefficients with Ridge Regression, by calculating the SE of the beta estimates after I choose the right lambda. ...
mslick3's user avatar
  • 31
3 votes
2 answers
3k views

K-fold Cross Validation and Training/CV/Test set Techniques for choosing regularization parameter of Regression

Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods: K-fold Cross Validation (...
hans-t's user avatar
  • 569
3 votes
1 answer
415 views

Why parameters become zero in Group Lasso

I've already studied about Ridge, Lasso and Group Lasso. Lasso can estimate essential parameters. That is, some parameters got zero. By the way, in group lasso, why parameters corresponding to a ...
kiwamizamurai's user avatar
2 votes
0 answers
38 views

How is the weight vector calculated when using kernel trick for ridge regression

Im trying to understand how kernelized ridge regression works, and how we manage to first transform, and subsequently learn on higher-dimensional features without explicitly having to calculate them. ...
pyrrosk's user avatar
  • 33
2 votes
0 answers
44 views

Distribution of Penalized Regression Coefficients

For both linear and logistic regression we know that the coefficient vector $\hat\beta$ holds an asymptotic normal distribution, therefore the the distribution of the linear predictor $\hat\theta_i=x^...
Spätzle's user avatar
  • 4,002
2 votes
0 answers
38 views

Coordinate Descent Alternating between LASSO and Ridge

Is there a way to do Coordinate descent but depending on the variable change the method applied to find the coefficient? For example, apply a LASSO constraint to a predefined 3 variables and Ridge to ...
Tylerr's user avatar
  • 1,552
2 votes
0 answers
65 views

If L2-Regularization includes no bias, why do many images show a circle as the constraint region?

I got a little bit (massively, to be honest), confused by the following apparent misconceptions I have learned recently. Looking for information about L2-Regularization, the following image is one of ...
kklaw's user avatar
  • 515
2 votes
0 answers
173 views

Intercept and slope of ridge regression model

When we compute a Ridge regression model, do we need to compute the intercept separately from the slopes? As you know, the estimated $\beta$ for the ridge regression model is given by: $\hat \beta = (...
Paca's user avatar
  • 121
2 votes
0 answers
131 views

Efficient computation of ridge/lasso regression

n the book "Introduction to Statistical learning with R" https://hastie.su.domains/ISLR2/ISLRv2_website.pdf , authors say on page 247 - "There are very efficient algorithms for fitting ...
Madhuresh's user avatar
2 votes
0 answers
698 views

Mixed model via ridge regression

A mixed model can be recast as a ridge regression for a specific regularization parameter $\lambda$ that penalizes only the random effects -- aka dummy variables for the grouping levels. Fitting a ...
generic_user's user avatar
  • 13.5k
2 votes
0 answers
580 views

Feature Selection via RFE with Ridge or SVM (Regression)

I have a regression problem where the number of samples n is less than the number of features p (e.g., p=500 and n=400; but the problem can be extended to p=3000+ and n=400). The features are largely ...
Pablo's user avatar
  • 43
2 votes
0 answers
97 views

Compare the MSE between LASSO and OLS

In regression, the MSE of estimation $\hat{\theta}$ is: $$MSE = E[(\hat{\theta} - \theta)^2].$$ I know the detailed comparison of MSE between OLS and Ridge. But can hardly find some materials between ...
user6703592's user avatar
  • 1,345
2 votes
0 answers
49 views

Can we estimate independent parameters when $p > n$?

I am using a ridge regression method to estimate the effect of SNPs (p = 10000) as random effect for a population of n=2000 individuals. I know that when we estimate fixed effects, the number of ...
marb_021's user avatar
2 votes
1 answer
425 views

Ridge regression and distribution of estimate?

When OLS overfits observed data, does it give skewed distribution of estimates?
Harshalkumar's user avatar
2 votes
0 answers
62 views

Effect of L2 regularization on Linear Regression

I am starting with L2 regularization on linear regression. Case without regularization: objective function $(Xw-y)^T(Xw-y)$ parameters vector $w= (X^TX)^{-1}X^Ty.$ Case with regularization: ...
Nabuko's user avatar
  • 31
2 votes
0 answers
225 views

How does ridge regression actually utilize principal components?

This is what I understand so far: Let $\mathbf X$ be the centered $n \times p$ predictor matrix and consider its singular value decomposition $\mathbf X = \mathbf{USV}^\top$ with $\mathbf S$ being a ...
Victor M's user avatar
  • 309
2 votes
0 answers
114 views

Is it correct to combine any feature selection (backward/forward/stepwise) with regularization in logistic regression?

I use stepwise regression for exclude "worst" features (based on p-value) and after try to build model with L2 regularization on selected features. Emperically, this model is better that ...
Timofey Vilkov's user avatar
2 votes
0 answers
318 views

Intuition behind the degrees of freedom in ridge regression

I'm reading through the ESL book and I'm on the part of ridge regression where the effective degrees of freedom are defined $$ df(\lambda) = tr(X(X'X + \lambda I)^{-1}X') = \sum_{j=1}^p{\frac{d_j^2}{...
Nikola's user avatar
  • 141

1
2 3 4 5