# Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

788 questions
Filter by
Sorted by
Tagged with
3k views

### K-fold Cross Validation and Training/CV/Test set Techniques for choosing regularization parameter of Regression

Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods: K-fold Cross Validation (...
13 views

### Optimisation of Polynomial Fittting Process

I have built a multitvariate log link GLM model and I want to fit polynomials to some of the numerical variates (i.e. fit polynomials of order 1,2,3 etc to the relativities of the model). However, I ...
8k views

### The results of CV on Ridge are different than the results of RidgeCV

I am using cross_val_predict to generate cross-validated estimates using Ridge Regression: ...
684 views

### Ridge regression, large lambda results in smaller RMSE of the training data

I am training the ridge regression on a one-day sensor data using the closed-form solution where $$\beta=(X^TX+\lambda*I)^{-1}X^TY$$ and Matlab. The $X$ is 15 polynomial time matrix. I created a ...
267 views

### How to "choose" binary variables which have a big impact on a regression?

I am currently facing an issue with analyzing my data for a project. I have a dataset of about 100.000 samples. I have approximate 50 columns which are all binary and my dependent variable is time ...
424 views

### Ridge regression and distribution of estimate?

When OLS overfits observed data, does it give skewed distribution of estimates?
107k views

### Why L1 norm for sparse models

I am reading books about linear regression. There are some sentences about the L1 and L2 norm. I know the formulas, but I don't understand why the L1 norm enforces sparsity in models. Can someone give ...
1 vote
70 views

### Can I utilize Ridge Regression to update coefficients of a Linear Regression model for a new dataset?

I have fitted a Linear Regression Model using one dataset. Now, I have another smaller dataset that I want to refine the model with. Can I use Ridge regression to update the estimated coefficients for ...
252 views

### Should interactions also be scaled in LASSO/Ridge, or just constituent covariates?

I understand that in LASSO/Ridge it is best practice to scale covariates so that no single covariate dominates the penalized norm. However, when entering interaction terms, it is unclear whether only ...
4k views

### Ridge penalized GLMs using row augmentation?

I've read that ridge regression could be achieved by simply adding rows of data to the original data matrix, where each row is constructed using 0 for the dependent variables and the square root of $k$...
7 views

### Computing Test Loss in Kernel Ridge Regression

In Kernel Ridge regression we have the standard loss function $$L(\beta) = \|Y-K\beta\|_2^2 + \alpha \beta^T K \beta$$ Here, $K$ is the kernel (gram) matrix. If I compute $\beta$ on a training set, so ...
21 views

29 views

### Does solution to ridge regression still minimizes the cost function when lambda is <=0?

This was a homework problem where I was asked to find explicit expression that minimises the cost function. I found the solution as : $\hat{\theta} = (X^TX + \lambda I)^{-1}X^Ty$ Now the problem ...
20 views

1k views

### Why not use Ridge after Lasso vs relaxed Lasso

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
76 views

### Fixed-effect model with ridge regression, or how else to deal with multicollinearity

I am currently writing a registered report for data which will be clustered within eight countries. Since that is too few to do a multilevel model with random effects (McNeish & Stapleton, 2016), ...
1 vote
97 views

### Multicollinearity and large OLS estimates vs ridge regression

The point of regularization methods (for example ridge regression) is to penalize large ordinary least squares estimates. We know that variance-covariance matrix for OLS estimates can be decomposed ...
32 views

### Intuition for how individual coefficients change with increasing regularization penalties [duplicate]

I'm trying to build intuition around how individual coefficients change as a regularization penalty is increased (for both ridge and lasso). This is what I understand the curves of the l1 and l2 ...
5k views

### Maximum penalty for ridge regression

Consider a regression model $$y = X \beta + \varepsilon.$$ I will use ridge regression to estimate $\beta$. Ridge regression contains a tuning parameter (the penalty intensity) $\lambda$. If I ...
27 views

### For variable selection, would a viable alternative to using lasso be to use ridge with a threshold, or is switching to elastic net preferred?

A similar question was asked here Why can't ridge regression provide better interpretability than LASSO?, and the answer suggested that a main difference between lasso and ridge is that a zero ...
1 vote
95 views

### Robust way to add predictors to existing linear model

I'm looking for a robust way to gradually build up a regression model -- namely I have a linear base-model with a robust set of predictors for which I'm fairly certain I have near optimal weights for, ...
1 vote
64 views

### What is the objective function for weighted lasso & ridge?

For weighted OLS, the objective function can be written as $$\arg \min_{\beta} ||W^{0.5}(y - X\beta)||^2$$ This is quite similar to the objective function for plain OLS, except without the $W$ term: ...
684 views

### Why regularization/shrinkage method works for p>n?

I am having trouble visualizing regularization/shrinkage method for the case of p>n. If I have only two data point, but I want to fit a plane ($y = \beta_0+\beta_1x_1+\beta_2x_2+\epsilon$) through ...
1 vote
8 views

### Relationship between the t-statistic of a coefficient in an OLS multivariate regression and Ridge shrinkage?

If I'm running a multivariate OLS regression and look at the t-stats of coefficients, is it the case that the coefficients with smaller t-stats are shrunk relatively more if I were to run the same ...
1 vote
21 views

### Understanding application Lasso and Ridge Regression

Currently reading up on Ridge and Lasso regression, have some questions to clarify. Suppose Model 1 has all predictors (i.e., 8) and Model 2 only has a specific subset chosen after EDA (i.e., 5) ...
28 views

### weird lasso prediction when using lambda 1se

I have performed a leave-one out cross-validated prediction using a lasso regression (with both lambda min and lambda 1se). My sample size is 52 and I have a bit more than 20 predictors. While lambda ...
34 views

### Is using VIF to Select Lambda in Ridge Regression a valid approach?

I recently came across an article that suggests selecting the lambda parameter in ridge regression based on Variance Inflation Factor (VIF) values. The method aims to choose a lambda that ensures all ...
63 views

### How to get Predicted Value in a ridge regression?

How to get Predicted Value from a Ridge regression using closed solution? I know that by applying the we get the vector of coefficients, but do we do next?
212 views

### In a Ridge regression, why do i get a stronger shrinkage when i remove some coefficients from the penalization term?

I cannot understand why in a ridge regression if I remove some coefficients from the penalty term I have a stronger shrinkage of the remaining coefficients that are included in the penalty term. From ...
1k views

### Too good to be true? Ridge prediction

I have a small data set of 18 persons. I have an outcome variable Y, and 200 predictors. These predictors were chosen based on biology and prior data. I used the caret R package and split the data set ...
584 views

### Standardization in penalized regression using glmnet

I want to run a penalized multinomial logit and logit regression using the glmnet package in R. I understand, that before fitting the penalized model, one should ...
68 views

### Correspondence between augmented design matrices and modified loss functions in linear regression?

Background Exercise 3.12 of "Elements of Statistical Learning" by by Hastie, Tibshirani, and Friedman reads as follows: Show that the ridge regression estimates can be obtained by ordinary ...