Ridge Regression is a technique which penalizes the size of regression coefficients in order to deal with multicollinear variables or ill-posed statistical problems.

learn more… | top users | synonyms

2
votes
1answer
55 views

Linear Regression with $L_2$: Different penalty strengths yield the same parameters?

Suppose I have model 1: $$Y=aX$$ where $X$ is $n\times1$, consisting of a single feature. suppose I fit this model with $L_2$ penalty with coefficient $\lambda=1$ : $$ \underset{a}{\min} ...
6
votes
1answer
61 views

What's the typical range of possible values for the shrinkage parameter in penalized regression?

In lasso or ridge regression, one has to specify a shrinkage parameter, often called by $\lambda$ or $\alpha$. This value is often chosen via cross validation by checking a bunch of different values ...
1
vote
1answer
35 views

Bias correction of ridge regression

I'm using ridge regression for estimating the fair value of variable $Y$ against a vector of correlated predictor variables $X_i$, based on past observations of both $Y$ and $X_i$. Let's assume that ...
1
vote
1answer
36 views

How to Find Adjusted $R^2$ or $R^2$ from Lasso and Ridge regression model

How do I find the adjusted $R^2$ (or $r^2$) from Lasso and Ridge regression? I used the glmnet package. For instance if I have this code so far.... ...
0
votes
0answers
10 views

Explicit optimization of RIDGE penalization parameter

I work with a very simple setup: I have two possibly very correlated features $x_{1,2}$, with normaliazation such as $\text{var}(x_{1,2}) = 1$, $Ey = Ex_{1,2} = 0$ (for the empirical distribution). I ...
1
vote
1answer
31 views

Calculated breeding values using markers using animal model in R

Animal model (frequently used in animal science and sometime in human or plants) is mixed model with: $y$ = $X$$b$ + $Z$$u$ + $e$ y is observed values for any quantitative variable, $Xb$ is fixed ...
6
votes
1answer
71 views

How to decide which penalty measure to use ? any general guidelines or thumb rules out of textbook

A number of regularization measures are available in literatures, which is kind of confusing to beginners. The classical penalty is ridge by Hoerl & Kennard (1970,Technometrics 12, 55–67). ...
2
votes
1answer
122 views

large variables and low sample (p > n) problem: ridge , LASSO, PLS, PCR which is most suitable for predictions

I am trying see whether to go for ridge regression, LASSO or principal component regression (PCR) or Partial Least Squares (PLS) in a situation where there are large number of variables / features (p) ...
5
votes
2answers
161 views

AIC, BIC and GCV: what is best for making decision in penalized regression methods?

My general understanding is AIC deals with the trade-off between the goodness of fit of the model and the complexity of the model. $AIC =2k -2ln(L)$ $k$ = number of parameters in the model $L$ = ...
3
votes
1answer
96 views

comparing OLS, ridge and lasso

I am trying to compare OLR, ridge and lasso in my situation. I could calculate SE for OLR and lasso but not for ridge. The following is Prostrate data from ...
2
votes
1answer
158 views

how to calculate effective degrees of freedom in ridge regression in R

Here is example to workout: ...
4
votes
1answer
74 views

demonstration of benefits of ridge regression over ordinary regression

I can understand ridge regression is better than ordinary regression in case of multiple-collinarity. ...
1
vote
0answers
20 views

Should the ridge parameter qualitatively change the weights?

I'm running a Ridge regression with about 1200 parameters (and about 30000 datapoints). I noticed that for some values of ridge, the weights look qualitatively different beyond a certain point. In ...
3
votes
1answer
84 views

Variance-covariance matrix for ridge regression with stochastic $\lambda$

In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression ...
5
votes
3answers
304 views

K-fold or hold-out cross validation for ridge regression using R

I am working on cross-validation of prediction of my data with 200 subjects and 1000 variables. I am interested ridge regression as number of variables (I want to use) is greater than number of ...
2
votes
1answer
51 views

Why does Ridge Regression work well in the presence of multicollinearity?

I am learning about ridge regression and know that ridge regression tends to work better in the presence of multicollinearity. I am wondering why this is true? Either an intuitive answer or a ...
0
votes
2answers
41 views

cross validation in ridge regression for classification. regularization issue

I perform ridge regression for classification. To find regularization parameter I do K-fold cross-validation with classification accuracy as a measure. This gives me some $\lambda$, which I then use ...
2
votes
0answers
63 views

Why is least squares performing as well as ridge regression when there is multicollinearity?

I am learning about ridge regression, so I am implementing it in MATLAB as practice. However, I am having trouble finding a structure of data where ridge regression performs better than an ordinary ...
0
votes
0answers
28 views

Laplace errors and ridge regression

Thinking about a normal linear regression, penalized by the L1 norm. That is the lasso, is there any literature on median ridge regression?. That is the residuals are calculated from an L1 norm ...
1
vote
0answers
135 views

K-fold Cross Validation and Training/CV/Test set Techniques for choosing regularization parameter of Regression

Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods: K-fold Cross Validation ...
4
votes
2answers
320 views

Ridge regression – Bayesian interpretation

I have heard that ridge regression can be derived as the mean of a posterior distribution, if the prior is adequately chosen. Is the intuition that the constraints as set on the regression ...
7
votes
2answers
279 views

Selection of k knots in regression smoothing spline equivalent to k categorical variables?

I'm working on a predictive cost model where the patient's age (an integer quantity measured in years) is one of the predictor variables. A strong nonlinear relationship between age and risk of a ...
1
vote
2answers
104 views

Strict convexity of Ridge vs Convexity of LASSO

Is there any intuition why the ridge regression is strictly convex, while the LASSO is only convex? Does it have to do with the "corners" of the L1 regularization?
3
votes
1answer
305 views

Ridge, lasso and elastic net

How do ridge, LASSO and elasticnet regularization methods compare? What are their respective advantages and disadvantages? Any good technical paper, or lecture notes would be appreciated as well.
1
vote
0answers
29 views

Difference between Primal, Dual and Kernel Ridge Regression

I would like to basically ask what the title says. What is the difference between Primal, Dual and Kernel Ridge Regression? People are using all three, and because of the different notation that ...
1
vote
0answers
40 views

Ridge regression in Matlab produces different results than direct calculation [duplicate]

I am trying to run Ridge regression in Matlab (ridge): $L$ is some matrix, $x$ is some random vector and $y=Lx+\alpha n$ is another vector. I expect ...
1
vote
0answers
35 views

How to calculate regularization parameter in generalized ridge regression given the degrees of freedom and input matrix?

I read a Q/A from here which is extremely nice. In the above Q/Answer the tuning parameter $λ$ was a scalar. But in my problem it is a vector $\lambda$ for a generalized ridge regression case. The ...
2
votes
1answer
211 views

Gaussian Process Kernel and Ridge Regression

Can a Dual Ridge Regression produce the same prediction results as a Gaussian Process with a polynomial kernel $K(x,x')=(x^Tx'+1)^2$ in less time complexity (GP is $O(n^3)$ ) using Cholesky ...
0
votes
0answers
43 views

What can be a cause of a extremely high standard coefficient?

I am using RapidMiner to perform linear regression with ridge parameter 1$\text{E}$ -8, min tolerance 0.05 and M5 prime for feature elimination. The std coefficient ...
3
votes
1answer
66 views

Equation 3.49 from Elements of Statistical Learning

I was on page 66 of ESL. I don't know how equation 3.49 on that page is derived. Where does the $N$ in the denominator come from ? Can someone kindly show me the steps in between the lines of equation ...
0
votes
0answers
33 views

On distance between parameters in Ridge regression [duplicate]

In ridge regression we know that as an estimate of $\beta$ and this gives the minimum sum of squares of the residuals: And we know that The question is how to demonstrate that
1
vote
0answers
44 views

Why does not ridge regression perform feature selection although it makes use of regularization?

Could somebody explain why ridge regression does not perform feature selection although it makes use of regularization? So, it penalizes the regression coefficients like LASSO does, but how come we ...
0
votes
0answers
99 views

Estimating Bias in a Linear Regression Model

Is there any way to estimate the bias of the estimate of the betas in a linear regression model when the actual beta values are unknown? The well known Mean Square Error (MSE) criterion is used to ...
1
vote
1answer
204 views

Fastest way to run ridge regression on large datasets where n>>p

Provided that you don't want to do any variable selection: Is there any software which is faster than glmnet at vanilla ridge regression for large datasets?
1
vote
2answers
98 views

Linear regression for time-series prediction

Say we have $N$ time series $X_t^i$ for $i=1...N$and we want to predict a separate time series $Y_t$. Let's consider the following model: $Y_t = \sum_{i} \beta_i X_{t-1}^i $ I am just trying to ...
2
votes
1answer
173 views

Applying ridge regression for an underdetermined system of equations?

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$ can be written as \begin{equation} \begin{array} &\operatorname{min}\ \| y - ...
3
votes
0answers
658 views

Confusion with Vowpal Wabbit's multiple-pass behavior when performing ridge-regression

I have encountered many peculiarities/misunderstandings of Vowpal Wabbit when trying to do online multiple-pass learning. Specifically, I need to solve a Ridge Linear regression problem, with ...
0
votes
0answers
43 views

Linearized Ridge Regression Estimator Under the Mean Squared Error Criterion in a Linear Regression Model

I have been implementing this paper: http://www.tandfonline.com/doi/abs/10.1080/03610918.2011.575506#.UrijOLS_UhI I can send a copy should anybody require it. I think that my implementation is ...
0
votes
0answers
48 views

How to estimate parameter for Biased Regression Estimator

Al-Gereari 2012 introduces the Modified Marquardt Estimator (MME) as: $\hat{\beta^{(r)}(m,k,J)}=T_r[\Delta_r(k)]^{-1}T_r'+m T_{p-r}[\Delta_{p-r}(k)]^{-1}T_{p-r}')(X'Y+kJ)$ where ...
1
vote
1answer
140 views

Coefficients from a penalized Cox PH

I'm using the R package Penalized (0.9-42) on a Cox PH model. I'm using L2 (Ridge) on the grounds that I don't want to shrink my coefficients to 0. I don't understand why when I ask for: ...
0
votes
1answer
107 views

Cross Validation for Ridge Regression

I'm using ridge regression for calculating optimal weights of a set of scores. These scores are correlated so the usage of ridge regression is used for penalizing large values of weights. So the ...
0
votes
2answers
139 views

Ridge Regression in R where coefficients are penalized toward numbers other than zero

Is it possible to penalize coefficients toward a number other than zero in a ridge regression in R? For example, let's say I have dependent variable Y and independent variables X1,X2,X3, and X4. ...
0
votes
0answers
45 views

Can't replicate the results of Golub's Generlized Cross Validation procedure (GCV)

I'm trying to understand this paper: Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter Author(s): Gene H. Golub, Michael Heath and Grace Wahba Source: Technometrics, Vol. ...
1
vote
0answers
78 views

Ridge regression not appropriate for collinearity caused by mathematical constraints on the data

In this paper: Use of the Bootstrap and Cross-Validation in Ridge Regression Author(s): Nancy Jo Delaney and Sangit Chatterjee Source: Journal of Business & Economic Statistics, Vol. 4, No. 2 ...
0
votes
0answers
50 views

Correct degrees of Freedom for Variance estimator in Ridge Regression

When I run a ridge regression on the following data using the Hoerl, Kennard and Baldwin estimator I only get the correct value of k when I use (n-p) where n is the number of observations and p the ...
1
vote
0answers
150 views

Pros and Cons of different types of regression? [closed]

I'm currently studying Linear Regression and am wondering if somebody would help me out with a high-level comparison of different methods. Just to list some: Lasso, Ridge, Elastic Net, Principal ...
0
votes
1answer
233 views

Ridge regression results different in using lm.ridge and glmnet

I applied some data to find the best variables solution of regression model using ridge regression in R. I have used lm.ridge and ...
4
votes
1answer
141 views

Large scale ridge regression

I'm trying to solve a problem of the form $\min_x \frac{1}{2}||Ax-b||^2_2 + \frac{\rho}{2}||x-z||^2_F$ where both $x$ and $b$ are high dimensional, and $b$ is much higher dimensional than $x$. The ...
2
votes
1answer
46 views

Simulating new x's in regression simulation study

One of my homework problems is a simulation that compares three estimators (least squares, ridge regression with known parameters, and ridge regression with estimated parameters) for the following ...
3
votes
0answers
342 views

Estimators for linear regression when multicollinearity is present

I have a multicollinearity problem is a linear regression model and ridge regression was suggested as a solution. So I have spent quite some time researching different ridge regressors in the ...