Skip to main content

Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

Filter by
Sorted by
Tagged with
1 vote
1 answer
70 views

Can I utilize Ridge Regression to update coefficients of a Linear Regression model for a new dataset?

I have fitted a Linear Regression Model using one dataset. Now, I have another smaller dataset that I want to refine the model with. Can I use Ridge regression to update the estimated coefficients for ...
Adham Enaya's user avatar
0 votes
0 answers
7 views

Computing Test Loss in Kernel Ridge Regression

In Kernel Ridge regression we have the standard loss function $$L(\beta) = \|Y-K\beta\|_2^2 + \alpha \beta^T K \beta$$ Here, $K$ is the kernel (gram) matrix. If I compute $\beta$ on a training set, so ...
WeakLearner's user avatar
  • 1,481
0 votes
0 answers
20 views

Lasso Regression Problem [duplicate]

$\operatorname*{argmin}_\beta\{\|y-X\beta\|^2 + \lambda\|\beta\|_1$, where $X$ is orthonormal. $\beta \in \mathbb R^d$. $X = [x_1,\ldots,x_n]^T$ and $y=(y_1,\ldots,y_n)^T \in \mathbb R^n$. $X^TX=I_{d\...
Harry Lofi's user avatar
0 votes
0 answers
40 views

Explicit form of L2 regularization in sklearn.linear_model.LogisticRegressionCV [duplicate]

I am using LogisticRegressionCV of sklearn, and I would like to know the explicit form of the L2 regularization in Logistic Regression. In the official page of LogisticRegressionCV, it is written $Cs$ ...
HQMA's user avatar
  • 145
0 votes
0 answers
18 views

How to sample with the 1-norm?

I am currently working on ridge regression, which can be interpreted using Bayesian statistics (DOI: 10.1016/j.electacta.2015.03.123). In particular, I know that the maximum-a-posteriori (MAP) ...
user avatar
2 votes
1 answer
29 views

Does solution to ridge regression still minimizes the cost function when lambda is <=0?

This was a homework problem where I was asked to find explicit expression that minimises the cost function. I found the solution as : $\hat{\theta} = (X^TX + \lambda I)^{-1}X^Ty$ Now the problem ...
aroma's user avatar
  • 123
0 votes
0 answers
20 views

Deriving a design Matrix for penalized regression [duplicate]

I am having issues attempting to derive this new design matrix. The objective function for the previous question was as follows: $\sum_{i}^{n}(Y_{i}-\mu)^2+\lambda\mu^2$ Find a design matrix $X(\...
Harry Lofi's user avatar
3 votes
0 answers
28 views

Effective degrees of freedom for residual variance in ridge regression

The definition of the effective degrees of freedom (dof) in Ridge Regression via the trace of the "hat matrix" is well known (see e.g. Hastie and Tibshirani's Generalized Additive Models). ...
Markus Loecher's user avatar
0 votes
0 answers
12 views

The relationship between ridge regularization and CNN Data Augmentation

In Chapter 10.3.4 of Introduction to Statistical Learning with Applications in Python by James et al. there is a sentence on data augmentation for CNNs (adding natural transformations of images into ...
pvelayudhan's user avatar
1 vote
0 answers
43 views

Why does ridge regression apply a non-monotone transformation to the singular values of the design matrix?

Per Wikipedia, Ridge Regression is equivalent to transforming the singular values $\sigma_i$ of the design matrix to $\frac{\sigma_i^2 + \alpha^2}{\sigma_i}$, where $\alpha$ is (in Wikipedia's ...
kuzzooroo's user avatar
  • 121
2 votes
0 answers
38 views

How is the weight vector calculated when using kernel trick for ridge regression

Im trying to understand how kernelized ridge regression works, and how we manage to first transform, and subsequently learn on higher-dimensional features without explicitly having to calculate them. ...
pyrrosk's user avatar
  • 33
2 votes
1 answer
140 views

What are a priori advantages of Lasso regularization for linear regression models?

What are a priori advantages of Lasso regularization for linear regression models, over many other heuristically-justifiable methods that both regularize the problem and perform variable selection? ...
AL1117's user avatar
  • 31
3 votes
1 answer
94 views

Dual form of the least square solution (ridge rigression)

I was reading this introductory material and on the 5th page, it describes the dual form of the least-square solution (with ridge regression) as $$A(aI + A^\top A)^{-1} = (aI + AA^\top)^{-1}A$$ for a $...
Alemu's user avatar
  • 125
0 votes
1 answer
76 views

Fixed-effect model with ridge regression, or how else to deal with multicollinearity

I am currently writing a registered report for data which will be clustered within eight countries. Since that is too few to do a multilevel model with random effects (McNeish & Stapleton, 2016), ...
Maximilian's user avatar
1 vote
2 answers
97 views

Multicollinearity and large OLS estimates vs ridge regression

The point of regularization methods (for example ridge regression) is to penalize large ordinary least squares estimates. We know that variance-covariance matrix for OLS estimates can be decomposed ...
Adam Bogdański's user avatar
3 votes
0 answers
27 views

For variable selection, would a viable alternative to using lasso be to use ridge with a threshold, or is switching to elastic net preferred?

A similar question was asked here Why can't ridge regression provide better interpretability than LASSO?, and the answer suggested that a main difference between lasso and ridge is that a zero ...
another_student's user avatar
0 votes
1 answer
32 views

Intuition for how individual coefficients change with increasing regularization penalties [duplicate]

I'm trying to build intuition around how individual coefficients change as a regularization penalty is increased (for both ridge and lasso). This is what I understand the curves of the l1 and l2 ...
another_student's user avatar
1 vote
1 answer
63 views

What is the objective function for weighted lasso & ridge?

For weighted OLS, the objective function can be written as $$ \arg \min_{\beta} ||W^{0.5}(y - X\beta)||^2 $$ This is quite similar to the objective function for plain OLS, except without the $W$ term: ...
24n8's user avatar
  • 1,147
1 vote
1 answer
95 views

Robust way to add predictors to existing linear model

I'm looking for a robust way to gradually build up a regression model -- namely I have a linear base-model with a robust set of predictors for which I'm fairly certain I have near optimal weights for, ...
ron burgundy's user avatar
1 vote
0 answers
8 views

Relationship between the t-statistic of a coefficient in an OLS multivariate regression and Ridge shrinkage?

If I'm running a multivariate OLS regression and look at the t-stats of coefficients, is it the case that the coefficients with smaller t-stats are shrunk relatively more if I were to run the same ...
user3089215's user avatar
1 vote
0 answers
21 views

Understanding application Lasso and Ridge Regression

Currently reading up on Ridge and Lasso regression, have some questions to clarify. Suppose Model 1 has all predictors (i.e., 8) and Model 2 only has a specific subset chosen after EDA (i.e., 5) ...
scoosch's user avatar
  • 11
0 votes
0 answers
28 views

weird lasso prediction when using lambda 1se

I have performed a leave-one out cross-validated prediction using a lasso regression (with both lambda min and lambda 1se). My sample size is 52 and I have a bit more than 20 predictors. While lambda ...
Simon's user avatar
  • 1
0 votes
0 answers
34 views

Is using VIF to Select Lambda in Ridge Regression a valid approach?

I recently came across an article that suggests selecting the lambda parameter in ridge regression based on Variance Inflation Factor (VIF) values. The method aims to choose a lambda that ensures all ...
HiddenLeafCoder's user avatar
2 votes
1 answer
62 views

How to get Predicted Value in a ridge regression?

How to get Predicted Value from a Ridge regression using closed solution? I know that by applying the we get the vector of coefficients, but do we do next?
ak3's user avatar
  • 21
0 votes
1 answer
211 views

In a Ridge regression, why do i get a stronger shrinkage when i remove some coefficients from the penalization term?

I cannot understand why in a ridge regression if I remove some coefficients from the penalty term I have a stronger shrinkage of the remaining coefficients that are included in the penalty term. From ...
SupBro's user avatar
  • 35
2 votes
1 answer
68 views

Correspondence between augmented design matrices and modified loss functions in linear regression?

Background Exercise 3.12 of "Elements of Statistical Learning" by by Hastie, Tibshirani, and Friedman reads as follows: Show that the ridge regression estimates can be obtained by ordinary ...
Steven Gubkin's user avatar
2 votes
2 answers
1k views

Too good to be true? Ridge prediction

I have a small data set of 18 persons. I have an outcome variable Y, and 200 predictors. These predictors were chosen based on biology and prior data. I used the caret R package and split the data set ...
user2862862's user avatar
0 votes
0 answers
20 views

Likelihood based CI with L2 regularization

I apologize if my question seems basic, but I'm attempting to derive confidence intervals for certain parameters whose estimates were obtained through nonlinear least squares regression. Unfortunately,...
Stefano Giampiccolo's user avatar
2 votes
0 answers
44 views

Distribution of Penalized Regression Coefficients

For both linear and logistic regression we know that the coefficient vector $\hat\beta$ holds an asymptotic normal distribution, therefore the the distribution of the linear predictor $\hat\theta_i=x^...
Spätzle's user avatar
  • 3,972
0 votes
0 answers
22 views

Ridge regression gives me this plot. How to interpret it? [duplicate]

I have done this plot with cv.glmnet(), can someone help me to interpret it? I also noticed that I get 2 different lambdas: lambda.min and lambda.1se What is the difference between these lambdas? Why ...
Jennie's user avatar
  • 1
2 votes
0 answers
38 views

Coordinate Descent Alternating between LASSO and Ridge

Is there a way to do Coordinate descent but depending on the variable change the method applied to find the coefficient? For example, apply a LASSO constraint to a predefined 3 variables and Ridge to ...
Tylerr's user avatar
  • 1,552
0 votes
1 answer
78 views

Using Ridge Regression to estimate importance of multicollinear variables in python

New to statistical analysis so bear with me. I have a dataset with 1 (say y) dependent variable and 5 independent variables (say x1,x2,x3,x4,x5) which are highly correlated. I know that y depends (...
Tom Waits's user avatar
  • 101
0 votes
0 answers
87 views

What's the effect of doubling the data and copying the data for Lasso Regression and Ridge Regression on standard error?

Suppose we have a dataset $X$, where each piece data of $X$ is a row vector, and the data generation process satisfies Gaussian-Markov assumption. If we do ridge regression on $Y\sim 2X$, how does the ...
vyjtkbyykyhuk's user avatar
1 vote
0 answers
15 views

Increasing accuracy of prediction

I'm working with this data set trying to implement a model to predict the variable normexam. I've used the following models on sklearn, adding dummies for categorical variables, and got the following ...
Melanie's user avatar
  • 133
0 votes
1 answer
51 views

Understanding Ridge regression for GAMLSS

I am trying to understand the mathematics behind Ridge regression for parametric GAMLSS. As I understand so far, it introduces a penalty term defines as $$\text{Penalty term}=\lambda \sum_{j=1}^{J_k} \...
Emma Kathrine Kokborg Iversen's user avatar
1 vote
0 answers
12 views

Can elasticnet ever select a different set of predictors than LASSO for a given lambda? [closed]

Since ridge regression can never penalize coefficients to zero, can elasticnet ever select a different set of predictors than LASSO for a given lambda?
Michael Webb's user avatar
  • 2,156
2 votes
0 answers
20 views

Lasso vs. Ridge Regression [duplicate]

The lasso coefficients are the ones that minimize $RSS+\lambda \Sigma_{j=1}^{p} |\beta_j|$ whereas the ridge regression coefficients those that minimize $RSS+\lambda \Sigma_{j=1}^{p} \beta_j^2$. I don'...
ColorStatistics's user avatar
0 votes
0 answers
37 views

Time complexity of Bayesian Ridge Regression

What is the training and inference time complexity of Bayesian Ridge Regression (e.g. as implemented in sklearn) in terms of the number of samples n and the number of features d?
PascalIv's user avatar
  • 829
0 votes
0 answers
51 views

Why does scaling of one predictor influence the coefficient estimates of other predictors in ridge regression?

In Introduction to Statistical Learning it is written The standard least squares coefficient estimates discussed in Chapter 3 are scale equivariant: multiplying $X_j$ by a constant $c$ simply leads ...
Mateusz's user avatar
  • 59
1 vote
0 answers
296 views

Best Datasets and Packages for Comparing LASSO, Elastic Net, and Ridge [closed]

I have been recently been working with the MASS, lars, and glmnet packages to study variable ...
YessuhYessuhYessuh's user avatar
1 vote
0 answers
26 views

Differences in Performance Between in MASS Package lm.ridge() and enet in elasticnet Package

A background: I am currently working with the 'elasticnet' package (elasticnet v.1.3) maintained by Hui Zou. This package was developed to accompany Hui Zou and Trevor Hastie's Statistical Society B ...
YessuhYessuhYessuh's user avatar
1 vote
1 answer
41 views

Machine learning regression using centered and scaled data

Imagine I have a large dataset of many variables and many observations. I would like to create a regression model to predict the values of new data. For the sake of ease, say I find a ridge regression ...
TerryStone's user avatar
3 votes
1 answer
100 views

Explaining Constant $\mathrm{corr}(\hat{y},y)$ in Ridge Regression as $\lambda$ Varies

I was recently asked the question, "In ridge regression $\hat{y}=X(X^\top X+\lambda I)^{-1} X^\top y$, why might the correlation $\mathrm{corr}(\hat{y},y)$ between the predicted values ($\hat{y}$)...
Alex's user avatar
  • 307
6 votes
0 answers
99 views

Generalize the 1SE rule to elastic net

When you do LASSO or ridge regression, and pick the hyperparameter using cross-validation, the 1SE rule suggest to select not the best CV result but the one with the most penalization that's still ...
Björn's user avatar
  • 32.8k
1 vote
0 answers
108 views

Quadratic Programming for ridge regression [closed]

I'm trying to use either pracma::quadprog or quadprog::solve.QP in R to solve a ridge regression, which can be written as a ...
alice123019's user avatar
4 votes
1 answer
88 views

Large sample limit of linear and ridge regression

I wanted to check if these reasonings are correct. The formula for a multilinear regression, input $X_{s,i}$, where $s$ is the sample and $i$ the features, and output $Y$, is given by: $$\beta=(X^+X)^{...
Thomas's user avatar
  • 890
2 votes
1 answer
142 views

Coefficient of highly correlated variables under LASSO and ridge

I have been presented with some interesting questions but unfortunately, I am struggling to provide satisfactory answers. The questions are as follows: How will the regression coefficients of two ...
Alex's user avatar
  • 307
3 votes
1 answer
303 views

Interpreting the biased coefficients of a ridge or LASSO regression model [duplicate]

In a recent conversation with one of the colleagues I was presented with a view that LASSO/Ridge regularization (trading bias for variance) renders coefficient estimates useless for interpretation, i....
Always Right Never Left's user avatar
2 votes
1 answer
90 views

Question about retraining a regression model

Exercise. Suppose you train a Ridge model to a regression problem that has a normalized perfomance measure (say K) that attains a value in the interval [0,1], where 0 means that the model is terrible ...
xyz's user avatar
  • 185
0 votes
0 answers
69 views

Ridge Estimator in Summation Form

I am trying to derive $\widehat{\beta}$ in summation form from the following: $$\text{argmin } \sum_{i=1}^{N}(y_i - X_i^{T}\beta)^2 + \lambda \sum_{k=1}^{K}{\beta}_k^2$$ I do not want to resort to ...
Kam's user avatar
  • 101

1
2 3 4 5
16