Skip to main content

Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

Filter by
Sorted by
Tagged with
3 votes
2 answers
3k views

K-fold Cross Validation and Training/CV/Test set Techniques for choosing regularization parameter of Regression

Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods: K-fold Cross Validation (...
0 votes
0 answers
13 views

Optimisation of Polynomial Fittting Process

I have built a multitvariate log link GLM model and I want to fit polynomials to some of the numerical variates (i.e. fit polynomials of order 1,2,3 etc to the relativities of the model). However, I ...
2 votes
2 answers
8k views

The results of CV on Ridge are different than the results of RidgeCV

I am using cross_val_predict to generate cross-validated estimates using Ridge Regression: ...
0 votes
1 answer
684 views

Ridge regression, large lambda results in smaller RMSE of the training data

I am training the ridge regression on a one-day sensor data using the closed-form solution where $$ \beta=(X^TX+\lambda*I)^{-1}X^TY $$ and Matlab. The $X$ is 15 polynomial time matrix. I created a ...
0 votes
1 answer
267 views

How to "choose" binary variables which have a big impact on a regression?

I am currently facing an issue with analyzing my data for a project. I have a dataset of about 100.000 samples. I have approximate 50 columns which are all binary and my dependent variable is time ...
2 votes
1 answer
424 views

Ridge regression and distribution of estimate?

When OLS overfits observed data, does it give skewed distribution of estimates?
156 votes
8 answers
107k views

Why L1 norm for sparse models

I am reading books about linear regression. There are some sentences about the L1 and L2 norm. I know the formulas, but I don't understand why the L1 norm enforces sparsity in models. Can someone give ...
1 vote
1 answer
70 views

Can I utilize Ridge Regression to update coefficients of a Linear Regression model for a new dataset?

I have fitted a Linear Regression Model using one dataset. Now, I have another smaller dataset that I want to refine the model with. Can I use Ridge regression to update the estimated coefficients for ...
4 votes
1 answer
252 views

Should interactions also be scaled in LASSO/Ridge, or just constituent covariates?

I understand that in LASSO/Ridge it is best practice to scale covariates so that no single covariate dominates the penalized norm. However, when entering interaction terms, it is unclear whether only ...
14 votes
3 answers
4k views

Ridge penalized GLMs using row augmentation?

I've read that ridge regression could be achieved by simply adding rows of data to the original data matrix, where each row is constructed using 0 for the dependent variables and the square root of $k$...
0 votes
0 answers
7 views

Computing Test Loss in Kernel Ridge Regression

In Kernel Ridge regression we have the standard loss function $$L(\beta) = \|Y-K\beta\|_2^2 + \alpha \beta^T K \beta$$ Here, $K$ is the kernel (gram) matrix. If I compute $\beta$ on a training set, so ...
0 votes
0 answers
21 views

Lasso Regression Problem [duplicate]

$\operatorname*{argmin}_\beta\{\|y-X\beta\|^2 + \lambda\|\beta\|_1$, where $X$ is orthonormal. $\beta \in \mathbb R^d$. $X = [x_1,\ldots,x_n]^T$ and $y=(y_1,\ldots,y_n)^T \in \mathbb R^n$. $X^TX=I_{d\...
0 votes
0 answers
40 views

Explicit form of L2 regularization in sklearn.linear_model.LogisticRegressionCV [duplicate]

I am using LogisticRegressionCV of sklearn, and I would like to know the explicit form of the L2 regularization in Logistic Regression. In the official page of LogisticRegressionCV, it is written $Cs$ ...
13 votes
3 answers
1k views

Why l2 norm squared but l1 norm not squared?

In the Lasso, and ElasticNet, we use, as penalty, the l1 norm without squaring. But in the ElasticNet and Ridge, we use the l2 norm squared. Why is that, is there a particular reason (computational, ...
0 votes
0 answers
18 views

How to sample with the 1-norm?

I am currently working on ridge regression, which can be interpreted using Bayesian statistics (DOI: 10.1016/j.electacta.2015.03.123). In particular, I know that the maximum-a-posteriori (MAP) ...
3 votes
0 answers
29 views

Effective degrees of freedom for residual variance in ridge regression

The definition of the effective degrees of freedom (dof) in Ridge Regression via the trace of the "hat matrix" is well known (see e.g. Hastie and Tibshirani's Generalized Additive Models). ...
3 votes
1 answer
5k views

Mean Squared Error (MSE) of Ridge Regression

I am currently trying to understand the MSE of ridge regression. First, I am calculating the MSE mathematically, but I found it quite vague. After reviewing some books and articles I understood that $...
2 votes
1 answer
29 views

Does solution to ridge regression still minimizes the cost function when lambda is <=0?

This was a homework problem where I was asked to find explicit expression that minimises the cost function. I found the solution as : $\hat{\theta} = (X^TX + \lambda I)^{-1}X^Ty$ Now the problem ...
0 votes
0 answers
20 views

Deriving a design Matrix for penalized regression [duplicate]

I am having issues attempting to derive this new design matrix. The objective function for the previous question was as follows: $\sum_{i}^{n}(Y_{i}-\mu)^2+\lambda\mu^2$ Find a design matrix $X(\...
1 vote
1 answer
289 views

Kernel trick implemented for Ridge Regression

I am trying to see the kernel trick implemented for Ridge Regression. As a first step, I want to rewrite the solution of Ridge regression. I know that: $ \hat{\beta} = (X^TX + \lambda I_n)^{-1} X^T Y $...
0 votes
0 answers
12 views

The relationship between ridge regularization and CNN Data Augmentation

In Chapter 10.3.4 of Introduction to Statistical Learning with Applications in Python by James et al. there is a sentence on data augmentation for CNNs (adding natural transformations of images into ...
1 vote
0 answers
43 views

Why does ridge regression apply a non-monotone transformation to the singular values of the design matrix?

Per Wikipedia, Ridge Regression is equivalent to transforming the singular values $\sigma_i$ of the design matrix to $\frac{\sigma_i^2 + \alpha^2}{\sigma_i}$, where $\alpha$ is (in Wikipedia's ...
2 votes
1 answer
313 views

Mixed Model: Ridge Regression and Data Augmentation

Supposed I have a mixed model in the form: $$y = X\beta + Zu+ \varepsilon$$ If I want to enforce a constraint on the $\beta$s can I follow the data augmentation approach that @whuber mentioned here: ...
209 votes
4 answers
174k views

When should I use lasso vs ridge?

Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization ...
5 votes
3 answers
2k views

Ridge Regression in R where coefficients are penalized toward numbers other than zero

Is it possible to penalize coefficients toward a number other than zero in a ridge regression in R? For example, let's say I have dependent variable Y and independent variables X1,X2,X3, and X4. ...
2 votes
0 answers
38 views

How is the weight vector calculated when using kernel trick for ridge regression

Im trying to understand how kernelized ridge regression works, and how we manage to first transform, and subsequently learn on higher-dimensional features without explicitly having to calculate them. ...
2 votes
3 answers
5k views

Why are solution to ridge regression always expressed using matrix notation?

Consider the following ridge regression problem: minimize the loss function $$\sum_{i=1}^n ||y_i - w^T x_i||_2^2 + \lambda ||w||_2^2$$ with respect to the weight vector $w$. Taking derivative with ...
2 votes
1 answer
140 views

What are a priori advantages of Lasso regularization for linear regression models?

What are a priori advantages of Lasso regularization for linear regression models, over many other heuristically-justifiable methods that both regularize the problem and perform variable selection? ...
24 votes
1 answer
22k views

Difference between Primal, Dual and Kernel Ridge Regression

What is the difference between Primal, Dual and Kernel Ridge Regression? People are using all three, and because of the different notation that everyone uses at different sources is difficult for me ...
3 votes
1 answer
97 views

Dual form of the least square solution (ridge rigression)

I was reading this introductory material and on the 5th page, it describes the dual form of the least-square solution (with ridge regression) as $$A(aI + A^\top A)^{-1} = (aI + AA^\top)^{-1}A$$ for a $...
6 votes
1 answer
1k views

Why not use Ridge after Lasso vs relaxed Lasso

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
0 votes
1 answer
76 views

Fixed-effect model with ridge regression, or how else to deal with multicollinearity

I am currently writing a registered report for data which will be clustered within eight countries. Since that is too few to do a multilevel model with random effects (McNeish & Stapleton, 2016), ...
1 vote
2 answers
97 views

Multicollinearity and large OLS estimates vs ridge regression

The point of regularization methods (for example ridge regression) is to penalize large ordinary least squares estimates. We know that variance-covariance matrix for OLS estimates can be decomposed ...
0 votes
1 answer
32 views

Intuition for how individual coefficients change with increasing regularization penalties [duplicate]

I'm trying to build intuition around how individual coefficients change as a regularization penalty is increased (for both ridge and lasso). This is what I understand the curves of the l1 and l2 ...
12 votes
2 answers
5k views

Maximum penalty for ridge regression

Consider a regression model $$ y = X \beta + \varepsilon. $$ I will use ridge regression to estimate $\beta$. Ridge regression contains a tuning parameter (the penalty intensity) $\lambda$. If I ...
3 votes
0 answers
27 views

For variable selection, would a viable alternative to using lasso be to use ridge with a threshold, or is switching to elastic net preferred?

A similar question was asked here Why can't ridge regression provide better interpretability than LASSO?, and the answer suggested that a main difference between lasso and ridge is that a zero ...
1 vote
1 answer
95 views

Robust way to add predictors to existing linear model

I'm looking for a robust way to gradually build up a regression model -- namely I have a linear base-model with a robust set of predictors for which I'm fairly certain I have near optimal weights for, ...
1 vote
1 answer
64 views

What is the objective function for weighted lasso & ridge?

For weighted OLS, the objective function can be written as $$ \arg \min_{\beta} ||W^{0.5}(y - X\beta)||^2 $$ This is quite similar to the objective function for plain OLS, except without the $W$ term: ...
2 votes
2 answers
684 views

Why regularization/shrinkage method works for p>n?

I am having trouble visualizing regularization/shrinkage method for the case of p>n. If I have only two data point, but I want to fit a plane ($y = \beta_0+\beta_1x_1+\beta_2x_2+\epsilon$) through ...
1 vote
0 answers
8 views

Relationship between the t-statistic of a coefficient in an OLS multivariate regression and Ridge shrinkage?

If I'm running a multivariate OLS regression and look at the t-stats of coefficients, is it the case that the coefficients with smaller t-stats are shrunk relatively more if I were to run the same ...
1 vote
0 answers
21 views

Understanding application Lasso and Ridge Regression

Currently reading up on Ridge and Lasso regression, have some questions to clarify. Suppose Model 1 has all predictors (i.e., 8) and Model 2 only has a specific subset chosen after EDA (i.e., 5) ...
0 votes
0 answers
28 views

weird lasso prediction when using lambda 1se

I have performed a leave-one out cross-validated prediction using a lasso regression (with both lambda min and lambda 1se). My sample size is 52 and I have a bit more than 20 predictors. While lambda ...
0 votes
0 answers
34 views

Is using VIF to Select Lambda in Ridge Regression a valid approach?

I recently came across an article that suggests selecting the lambda parameter in ridge regression based on Variance Inflation Factor (VIF) values. The method aims to choose a lambda that ensures all ...
2 votes
1 answer
63 views

How to get Predicted Value in a ridge regression?

How to get Predicted Value from a Ridge regression using closed solution? I know that by applying the we get the vector of coefficients, but do we do next?
0 votes
1 answer
212 views

In a Ridge regression, why do i get a stronger shrinkage when i remove some coefficients from the penalization term?

I cannot understand why in a ridge regression if I remove some coefficients from the penalty term I have a stronger shrinkage of the remaining coefficients that are included in the penalty term. From ...
2 votes
2 answers
1k views

Too good to be true? Ridge prediction

I have a small data set of 18 persons. I have an outcome variable Y, and 200 predictors. These predictors were chosen based on biology and prior data. I used the caret R package and split the data set ...
3 votes
0 answers
584 views

Standardization in penalized regression using glmnet

I want to run a penalized multinomial logit and logit regression using the glmnet package in R. I understand, that before fitting the penalized model, one should ...
2 votes
1 answer
68 views

Correspondence between augmented design matrices and modified loss functions in linear regression?

Background Exercise 3.12 of "Elements of Statistical Learning" by by Hastie, Tibshirani, and Friedman reads as follows: Show that the ridge regression estimates can be obtained by ordinary ...
0 votes
0 answers
20 views

Likelihood based CI with L2 regularization

I apologize if my question seems basic, but I'm attempting to derive confidence intervals for certain parameters whose estimates were obtained through nonlinear least squares regression. Unfortunately,...
6 votes
0 answers
99 views

Generalize the 1SE rule to elastic net

When you do LASSO or ridge regression, and pick the hyperparameter using cross-validation, the 1SE rule suggest to select not the best CV result but the one with the most penalization that's still ...

1
2 3 4 5
16