Ridge Regression is a technique which penalizes the size of regression coefficients in order to deal with multicollinear variables or ill-posed statistical problems.

learn more… | top users | synonyms

1
vote
1answer
42 views

3D surface plot for least square & ridge regression

I'm very impressed by this plot: Why does ridge estimate become better than OLS by adding a constant to the diagonal? Does someone has any clue about how to plot this on R? I mean, how to get RSS ...
1
vote
0answers
22 views

What criterion are used to accept/reject hypotheses in ridge regression?

On what basis might one accept/reject a hypothesis when running ridge regression? For example, if I have five predictor variables as part of a ridge regression... what criterion would I use to accept ...
0
votes
1answer
22 views

Ridge regression - what is k=0?

I'm getting to know ridge regression and what to check my understanding quickly. I understand that k is the shrinkage parameter. If I'm reading off coefficients where k=0, is that equivalent to an ...
1
vote
1answer
59 views

Ridge & LASSO norms

this post folloms this one: Why does ridge estimate become better than OLS by adding a constant to the diagonal? Here is my question: As far as I know, ridge regularization uses a l2-norm (euclidean ...
2
votes
1answer
39 views

Support vector regression versus kernel ridge regression

I have a question concerning the difference between support vector regression and kernel regression. I will try to write down all the math so no misunderstandings arise (hopefully). Let's begin with ...
17
votes
2answers
465 views

Why does ridge estimate become better than OLS by adding a constant to the diagonal?

I understand that the ridge regression estimate is the $\beta$ that minimizes residual sum of square and a penalty on the size of $\beta$ $$\beta_{ridge} = (\lambda I_D + X'X)^{-1}X'y = ...
2
votes
1answer
55 views

Linear Regression with $L_2$: Different penalty strengths yield the same parameters?

Suppose I have model 1: $$Y=aX$$ where $X$ is $n\times1$, consisting of a single feature. suppose I fit this model with $L_2$ penalty with coefficient $\lambda=1$ : $$ \underset{a}{\min} ...
6
votes
1answer
69 views

What's the typical range of possible values for the shrinkage parameter in penalized regression?

In lasso or ridge regression, one has to specify a shrinkage parameter, often called by $\lambda$ or $\alpha$. This value is often chosen via cross validation by checking a bunch of different values ...
1
vote
1answer
38 views

Bias correction of ridge regression

I'm using ridge regression for estimating the fair value of variable $Y$ against a vector of correlated predictor variables $X_i$, based on past observations of both $Y$ and $X_i$. Let's assume that ...
1
vote
1answer
52 views

How to Find Adjusted $R^2$ or $R^2$ from Lasso and Ridge regression model

How do I find the adjusted $R^2$ (or $r^2$) from Lasso and Ridge regression? I used the glmnet package. For instance if I have this code so far.... ...
0
votes
0answers
10 views

Explicit optimization of RIDGE penalization parameter

I work with a very simple setup: I have two possibly very correlated features $x_{1,2}$, with normaliazation such as $\text{var}(x_{1,2}) = 1$, $Ey = Ex_{1,2} = 0$ (for the empirical distribution). I ...
1
vote
1answer
55 views

Calculated breeding values using markers using animal model in R

Animal model (frequently used in animal science and sometime in human or plants) is mixed model with: $y$ = $X$$b$ + $Z$$u$ + $e$ y is observed values for any quantitative variable, $Xb$ is fixed ...
6
votes
1answer
74 views

How to decide which penalty measure to use ? any general guidelines or thumb rules out of textbook

A number of regularization measures are available in literatures, which is kind of confusing to beginners. The classical penalty is ridge by Hoerl & Kennard (1970,Technometrics 12, 55–67). ...
2
votes
1answer
152 views

large variables and low sample (p > n) problem: ridge , LASSO, PLS, PCR which is most suitable for predictions

I am trying see whether to go for ridge regression, LASSO or principal component regression (PCR) or Partial Least Squares (PLS) in a situation where there are large number of variables / features (p) ...
6
votes
2answers
209 views

AIC, BIC and GCV: what is best for making decision in penalized regression methods?

My general understanding is AIC deals with the trade-off between the goodness of fit of the model and the complexity of the model. $AIC =2k -2ln(L)$ $k$ = number of parameters in the model $L$ = ...
3
votes
1answer
122 views

comparing OLS, ridge and lasso

I am trying to compare OLR, ridge and lasso in my situation. I could calculate SE for OLR and lasso but not for ridge. The following is Prostrate data from ...
2
votes
1answer
219 views

how to calculate effective degrees of freedom in ridge regression in R

Here is example to workout: ...
4
votes
1answer
84 views

demonstration of benefits of ridge regression over ordinary regression

I can understand ridge regression is better than ordinary regression in case of multiple-collinarity. ...
1
vote
0answers
22 views

Should the ridge parameter qualitatively change the weights?

I'm running a Ridge regression with about 1200 parameters (and about 30000 datapoints). I noticed that for some values of ridge, the weights look qualitatively different beyond a certain point. In ...
3
votes
1answer
89 views

Variance-covariance matrix for ridge regression with stochastic $\lambda$

In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression ...
5
votes
3answers
403 views

K-fold or hold-out cross validation for ridge regression using R

I am working on cross-validation of prediction of my data with 200 subjects and 1000 variables. I am interested ridge regression as number of variables (I want to use) is greater than number of ...
2
votes
1answer
56 views

Why does Ridge Regression work well in the presence of multicollinearity?

I am learning about ridge regression and know that ridge regression tends to work better in the presence of multicollinearity. I am wondering why this is true? Either an intuitive answer or a ...
0
votes
2answers
53 views

cross validation in ridge regression for classification. regularization issue

I perform ridge regression for classification. To find regularization parameter I do K-fold cross-validation with classification accuracy as a measure. This gives me some $\lambda$, which I then use ...
2
votes
0answers
72 views

Why is least squares performing as well as ridge regression when there is multicollinearity?

I am learning about ridge regression, so I am implementing it in MATLAB as practice. However, I am having trouble finding a structure of data where ridge regression performs better than an ordinary ...
0
votes
0answers
31 views

Laplace errors and ridge regression

Thinking about a normal linear regression, penalized by the L1 norm. That is the lasso, is there any literature on median ridge regression?. That is the residuals are calculated from an L1 norm ...
1
vote
0answers
173 views

K-fold Cross Validation and Training/CV/Test set Techniques for choosing regularization parameter of Regression

Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods: K-fold Cross Validation ...
4
votes
2answers
379 views

Ridge regression – Bayesian interpretation

I have heard that ridge regression can be derived as the mean of a posterior distribution, if the prior is adequately chosen. Is the intuition that the constraints as set on the regression ...
7
votes
2answers
320 views

Selection of k knots in regression smoothing spline equivalent to k categorical variables?

I'm working on a predictive cost model where the patient's age (an integer quantity measured in years) is one of the predictor variables. A strong nonlinear relationship between age and risk of a ...
1
vote
2answers
106 views

Strict convexity of Ridge vs Convexity of LASSO

Is there any intuition why the ridge regression is strictly convex, while the LASSO is only convex? Does it have to do with the "corners" of the L1 regularization?
4
votes
1answer
355 views

Ridge, lasso and elastic net

How do ridge, LASSO and elasticnet regularization methods compare? What are their respective advantages and disadvantages? Any good technical paper, or lecture notes would be appreciated as well.
1
vote
0answers
34 views

Difference between Primal, Dual and Kernel Ridge Regression

I would like to basically ask what the title says. What is the difference between Primal, Dual and Kernel Ridge Regression? People are using all three, and because of the different notation that ...
1
vote
0answers
40 views

Ridge regression in Matlab produces different results than direct calculation [duplicate]

I am trying to run Ridge regression in Matlab (ridge): $L$ is some matrix, $x$ is some random vector and $y=Lx+\alpha n$ is another vector. I expect ...
1
vote
0answers
38 views

How to calculate regularization parameter in generalized ridge regression given the degrees of freedom and input matrix?

I read a Q/A from here which is extremely nice. In the above Q/Answer the tuning parameter $λ$ was a scalar. But in my problem it is a vector $\lambda$ for a generalized ridge regression case. The ...
2
votes
1answer
237 views

Gaussian Process Kernel and Ridge Regression

Can a Dual Ridge Regression produce the same prediction results as a Gaussian Process with a polynomial kernel $K(x,x')=(x^Tx'+1)^2$ in less time complexity (GP is $O(n^3)$ ) using Cholesky ...
0
votes
0answers
46 views

What can be a cause of a extremely high standard coefficient?

I am using RapidMiner to perform linear regression with ridge parameter 1$\text{E}$ -8, min tolerance 0.05 and M5 prime for feature elimination. The std coefficient ...
3
votes
1answer
71 views

Equation 3.49 from Elements of Statistical Learning

I was on page 66 of ESL. I don't know how equation 3.49 on that page is derived. Where does the $N$ in the denominator come from ? Can someone kindly show me the steps in between the lines of equation ...
0
votes
0answers
33 views

On distance between parameters in Ridge regression [duplicate]

In ridge regression we know that as an estimate of $\beta$ and this gives the minimum sum of squares of the residuals: And we know that The question is how to demonstrate that
1
vote
0answers
49 views

Why does not ridge regression perform feature selection although it makes use of regularization?

Could somebody explain why ridge regression does not perform feature selection although it makes use of regularization? So, it penalizes the regression coefficients like LASSO does, but how come we ...
0
votes
0answers
106 views

Estimating Bias in a Linear Regression Model

Is there any way to estimate the bias of the estimate of the betas in a linear regression model when the actual beta values are unknown? The well known Mean Square Error (MSE) criterion is used to ...
1
vote
1answer
214 views

Fastest way to run ridge regression on large datasets where n>>p

Provided that you don't want to do any variable selection: Is there any software which is faster than glmnet at vanilla ridge regression for large datasets?
1
vote
2answers
101 views

Linear regression for time-series prediction

Say we have $N$ time series $X_t^i$ for $i=1...N$and we want to predict a separate time series $Y_t$. Let's consider the following model: $Y_t = \sum_{i} \beta_i X_{t-1}^i $ I am just trying to ...
2
votes
1answer
196 views

Applying ridge regression for an underdetermined system of equations?

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$ can be written as \begin{equation} \begin{array} &\operatorname{min}\ \| y - ...
3
votes
0answers
723 views

Confusion with Vowpal Wabbit's multiple-pass behavior when performing ridge-regression

I have encountered many peculiarities/misunderstandings of Vowpal Wabbit when trying to do online multiple-pass learning. Specifically, I need to solve a Ridge Linear regression problem, with ...
0
votes
0answers
47 views

Linearized Ridge Regression Estimator Under the Mean Squared Error Criterion in a Linear Regression Model

I have been implementing this paper: http://www.tandfonline.com/doi/abs/10.1080/03610918.2011.575506#.UrijOLS_UhI I can send a copy should anybody require it. I think that my implementation is ...
0
votes
0answers
48 views

How to estimate parameter for Biased Regression Estimator

Al-Gereari 2012 introduces the Modified Marquardt Estimator (MME) as: $\hat{\beta^{(r)}(m,k,J)}=T_r[\Delta_r(k)]^{-1}T_r'+m T_{p-r}[\Delta_{p-r}(k)]^{-1}T_{p-r}')(X'Y+kJ)$ where ...
1
vote
1answer
146 views

Coefficients from a penalized Cox PH

I'm using the R package Penalized (0.9-42) on a Cox PH model. I'm using L2 (Ridge) on the grounds that I don't want to shrink my coefficients to 0. I don't understand why when I ask for: ...
0
votes
1answer
113 views

Cross Validation for Ridge Regression

I'm using ridge regression for calculating optimal weights of a set of scores. These scores are correlated so the usage of ridge regression is used for penalizing large values of weights. So the ...
0
votes
2answers
155 views

Ridge Regression in R where coefficients are penalized toward numbers other than zero

Is it possible to penalize coefficients toward a number other than zero in a ridge regression in R? For example, let's say I have dependent variable Y and independent variables X1,X2,X3, and X4. ...
0
votes
0answers
45 views

Can't replicate the results of Golub's Generlized Cross Validation procedure (GCV)

I'm trying to understand this paper: Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter Author(s): Gene H. Golub, Michael Heath and Grace Wahba Source: Technometrics, Vol. ...
1
vote
0answers
79 views

Ridge regression not appropriate for collinearity caused by mathematical constraints on the data

In this paper: Use of the Bootstrap and Cross-Validation in Ridge Regression Author(s): Nancy Jo Delaney and Sangit Chatterjee Source: Journal of Business & Economic Statistics, Vol. 4, No. 2 ...