Ridge Regression is a technique which penalizes the size of regression coefficients in order to deal with multicollinear variables or ill-posed statistical problems.

learn more… | top users | synonyms

0
votes
0answers
14 views

Strange selection results, ridge logistic regression

I'm studying ridge logistic regression with glmnet on R. I have a lot of regressors which are dummies. I'm trying to maximise the AUC (prediction of a binary output). My question is about the graph ...
0
votes
1answer
18 views

MATLAB - using ridge regression weights

One simple and straightforward question, which is confusing me because poor results that I'm getting. I'm using MATLAB's built in ridge function to get weights for my model, on my training dataset. ...
0
votes
2answers
30 views

Can someone explain what the foldid argument in glmnet does?

I m trying to determine what alpha to use in my glmnet function, but the help file tells me: Note that cv.glmnet does NOT search for values for alpha. A ...
2
votes
0answers
25 views

Estimating the prediction variance in kernel ridge regression

I'm trying to estimate the variance of predictions for a kernel ridge regression model. The model is simply kernel ridge regression: $$\hat{y} = K(K+\lambda I)^{-1}y = A y$$ $K$ is the $n \times n$ ...
1
vote
1answer
56 views

How to calculate Hat matrix for penalized spline regressions?

The book "Semiparametric Regression" by Ruppert et al. (2003) provided a computationally fast algorithm for Penalized Spline Regression. I put a part of the algorithm here. Does anybody can do algebra ...
20
votes
2answers
215 views

confidence intervals' coverage with regularized estimates

Suppose I'm trying to estimate a large number of parameters from some high-dimensional data, using some kind of regularized estimates. The regularizer introduces some bias into the estimates, but it ...
2
votes
0answers
27 views

Is there a clear set of conditions under which lasso, ridge, or elastic net solution paths are monotone?

The question What to conclude from this lasso plot (glmnet) demonstrates solution paths for the lasso estimator that are not monotonic. That is, some of the cofficients grow in absolute value before ...
0
votes
2answers
94 views

Advantages and disadvantages of glmnet?

What are the advantages and disadvantages of glmnet with default options? glmnet(X,Y) In which situations it works best and in which situations it can be very ...
0
votes
0answers
50 views

When and when not to use ridge regression

What are the 'indications' (i.e. when to use) and 'contra-indications' (i.e. when not to use) of ridge-regression. I tried to read up on the net and it seems to useful when multi-collinearity is there ...
4
votes
4answers
189 views

How to interpret ridge regression plot

Following is the ridge regression example in MASS package: ...
0
votes
1answer
55 views

Extremly poor polynomial fitting with SVR in sklearn

I try to fit an obvious around degree 5 polynomial function. Much to my despair, sklearn bluntly refuses to match the polynomial, and instead output a 0-degree like ...
0
votes
1answer
67 views

How to calculate predicted values using an lm.ridge object?

I've found this line of code to calculate predicted values from a ridge.lm model: ...
3
votes
0answers
58 views

Regularization for ARIMA models

I am aware of LASSO, ridge and elastic-net type of regularization in linear regression models. Question: Can this (or a similar) kind of regularization be applied to ARIMA modelling (with a ...
1
vote
0answers
31 views

Does ridge regression adequately control for regression to the mean effects in my outcome variable?

I've been asked how to control for regression to the mean effects in a difference of differences analysis on health insurance data. We're measuring a utilization outcome both pre- and ...
0
votes
0answers
31 views

Survival analysis coxph using ridge regression with 2000 variables => “Penalty terms cannot be in an interaction”

I'm doing a survival analysis using coxph function (package survival). I'm using a ridge regression model included 2000 variables but I have a type of error message "...
1
vote
1answer
54 views

Sparsity in Lasso and advantage over ridge (Statistical Learning) [duplicate]

I'm learning about the Statistical learning and in the section comparing Lasso and Ridge Regression it shows that the main difference between these two problems is the way the constraint/penalty is ...
2
votes
1answer
54 views

How do I rank coefficients returned from a ridge regression?

I am running a ridge regression using GLMNET (alpha = 0) and would like to interpret the coefficients returned. I know there isn't really a significance test for this, but can I at least rank the ...
2
votes
2answers
237 views

Penalizing the Ordinary Least Squares estimation

In a regression analysis, we aim to find the best relationship between two variables (independent variable denoted $y$ and other dependent variable denoted by $x$, and which are related by: $y = ...
19
votes
2answers
856 views

Why is ridge regression called “ridge”, why is it needed, and what happens when $\lambda$ goes to infinity?

Ridge regression coefficient estimate $\hat{\beta}^R$ are the values that minimize the $$ \text{RSS} + \lambda \sum_{j=1}^p\beta_j^2. $$ My questions are: If $\lambda = 0$, then we see that the ...
0
votes
0answers
18 views

I did ridge regression and i am confused with coefficients

ridd=lm.ridge(mariner~o1+o2+o3,q,lambda=0.001) ridd o1 o2 o3 34.7597607381 0.0001989008 0.0393011905 ...
0
votes
0answers
42 views

My Cross-validation error is always increasing with increasing regularisation parameter

I am not sure what is happening, but my cross-validaton error is always increasing with increasing alpha in ridge regression. It should technically go down and then increase. Here is what I am doing ...
0
votes
2answers
44 views

How to find the best coefficient vector using cross-validation

So basically my dataset is divided into 5 train and 5 test folds. This is how I did in scikit: ...
1
vote
0answers
122 views

Ridge Regression STATA

I used ridge regression in order to dealing with multicollinearity but there is something that i do not understand. I use STATA command: -ridgereg y x1 x2 x3 x4 x5 x6 x7 x8 x9, ...
0
votes
1answer
47 views

For a quadratic form to minimize with a L2 regularization term, is the gradient of the solution collinear to the solution?

Say you minimize a quadratic form f with a L2 regularization term (g = f + L2_term). The solution of minimizing g is x*. Is the gradient of f applied to x* collinear to x* as the figure below ...
1
vote
0answers
24 views

How I Can Apply Ridge Regression? [closed]

my model is cross-sectional with macro variables and i run it with OLS. However, I found that it suffers from multicollinearity. I tried to change some variables but multicollinearity is still there. ...
4
votes
1answer
74 views

Range of lambda in elastic net regression

$\def\l{|\!|}$ Given the elastic net regression $$\min_b \frac{1}{2}\l y - Xb \l^2 + \alpha\lambda \l b\l_2^2 + (1 - \alpha) \lambda \l b\l_1$$ how can an appropriate range of $\lambda$ be chosen ...
0
votes
0answers
38 views

Getting very small values for lambda with glmnet()?

I'm using glmnet() to analyze a weather data set of 50 variables and 240 observations. My question is pretty simple: when I run ridge regression and LASSO on the ...
0
votes
3answers
54 views

Equating the two equations of Ridge Regression

I'm studying Ridge Regression now and I'm having a bit of trouble understanding how to relate the two equations that pop up when I read about it. There is the coefficient estimate: $$\hat{\beta} = ...
3
votes
1answer
68 views

Ridge regression in multivariate Gaussian distribution

When implementing GMM (Gaussian Mixture Model) in practice, the covariance matrix ${\Sigma}_{D\times D}$ is often singular. The reason is that we have to estimate $\frac{D(D+1)}{2}$ parameters in ...
0
votes
0answers
26 views

Hyper parameters optimization

Any one with a tip on how to define a suitable condition to find an optimal set of parameters for a combined norm regularization on a grid? Not cross validation or any related method. I am asking if ...
1
vote
0answers
56 views

Different behaviors for different Ridge implementations in R

I am having trouble reconciling the different behavior of different Ridge implementations in R. As the following code demonstrates it seems that MASS:lm.ridge ...
4
votes
1answer
125 views

Phoney data and ridge regression are the same?

I've read that ridge regression could be achieved by simply adding rows of data to the original data matrix, where each row is constructed using 0 for the dependent variables and the square root of ...
2
votes
0answers
23 views

Do mildly informative prior distributions tend to mitigate false positives (i.e. Type I error rates)?

I am curious if others have sources that speak to the matter that providing informative and/or mildly informative prior distributions on a parameter tend to mitigate false alarm rates? I know from the ...
2
votes
0answers
85 views

Why is intercept term left out of the penalty term in ridge regression? [duplicate]

I am reading about ridge regression in The Elements of Statistical Learning. In the ridge regression, we do not include intercept term $\beta_0$ in the penalty term. The book says, penalization of the ...
3
votes
0answers
61 views

Variance Inflation Factor less than 1 in ridge regression?

I was trying to determine the biasing constant in ridge regression when I came across a phenonomenon that seems quite puzzling, to me at least. I let the GCV criterion choose a constant for me and ...
15
votes
3answers
371 views

Interpretation of ridge regularization in regression

I have several questions regarding the ridge penalty in the least squares context: $$\beta_{ridge} = (\lambda I_D + X'X)^{-1}X'y$$ 1) The expression suggests that the covariance matrix of X is ...
4
votes
0answers
117 views

Why is glmnet giving me different answers than ridge estimation

I'm using glmnet to calculate ridge regression estimates. I got some results that made me suspicious in that glmnet is really doing what I think it does. To check this I wrote a simple R script where ...
3
votes
0answers
157 views

Finding standard error of beta coefficients in ridge regression using lambda

I need to get the standard errors of coefficients with Ridge Regression, by calculating the SE of the beta estimates after I choose the right lambda. ...
0
votes
0answers
54 views
0
votes
1answer
104 views

Have difficulty understanding Matlab's Ridge regression

I am confused by Matlab's documentation of Ridge regression at http://www.mathworks.com/help/stats/ridge-regression.html and couldn't figure it out by myself. On that page, the Introduction to Ridge ...
6
votes
1answer
327 views

Under exactly what conditions is ridge regression able to provide an improvement over ordinary least squares regression?

Ridge regression estimates parameters $\boldsymbol \beta$ in a linear model $\mathbf y = \mathbf X \boldsymbol \beta$ by $$\hat{\boldsymbol \beta}_\lambda = (\mathbf X^\top \mathbf X + \lambda \mathbf ...
0
votes
0answers
44 views

questions on glmnet result

I am trying to experiment with glmnet for a data set, which has 41 independent variables is 41. There are 80 data points in total. ...
28
votes
4answers
660 views

Unified view on shrinkage: what is the relation (if any) between Stein's paradox, ridge regression, and random effects in mixed models?

Consider the following three phenomena. Stein's paradox: given some data from multivariate normal distribution in $\mathbb R^n, \: n\ge 3$, sample mean is not a very good estimator of the true mean. ...
4
votes
1answer
85 views

Effective degrees of freedom for regularized regression

If I have a quadratic programming problem $$\min_b \frac{1}{2} b^tX^tXb - b^t(X^tY)$$ which I regularize by adding a multiple of the identity$\lambda I$ to $X^tX$, then the effective degrees of ...
2
votes
1answer
201 views

3D surface plot for least square & ridge regression

I'm very impressed by this plot: Why does ridge estimate become better than OLS by adding a constant to the diagonal? Does someone has any clue about how to plot this on R? I mean, how to get RSS ...
1
vote
1answer
98 views

What criteria are used to accept/reject hypotheses in ridge regression?

On what basis might one accept/reject a hypothesis when running ridge regression? For example, if I have five predictor variables as part of a ridge regression, what criterion would I use to accept ...
1
vote
1answer
40 views

Ridge regression - what is k=0?

I'm getting to know ridge regression and what to check my understanding quickly. I understand that k is the shrinkage parameter. If I'm reading off coefficients where k=0, is that equivalent to an ...
3
votes
2answers
197 views

Ridge & LASSO norms

This post follows this one: Why does ridge estimate become better than OLS by adding a constant to the diagonal? Here is my question: As far as I know, ridge regularization uses a l2-norm (euclidean ...
2
votes
1answer
230 views

Support vector regression versus kernel ridge regression

I have a question concerning the difference between support vector regression and kernel regression. I will try to write down all the math so no misunderstandings arise (hopefully). Let's begin with ...
27
votes
2answers
734 views

Why does ridge estimate become better than OLS by adding a constant to the diagonal?

I understand that the ridge regression estimate is the $\beta$ that minimizes residual sum of square and a penalty on the size of $\beta$ $$\beta_{ridge} = (\lambda I_D + X'X)^{-1}X'y = ...