Ridge Regression is a technique which penalizes the size of regression coefficients in order to deal with multicollinear variables or ill-posed statistical problems.

learn more… | top users | synonyms

7
votes
2answers
126 views

Selection of k knots in regression smoothing spline equivalent to k categorical variables?

I'm working on a predictive cost model where the patient's age (an integer quantity measured in years) is one of the predictor variables. A strong nonlinear relationship between age and risk of a ...
1
vote
2answers
71 views

Strict convexity of Ridge vs Convexity of LASSO

Is there any intuition why the ridge regression is strictly convex, while the LASSO is only convex? Does it have to do with the "corners" of the L1 regularization?
3
votes
1answer
138 views

Ridge, lasso and elastic net

How do ridge, LASSO and elasticnet regularization methods compare? What are their respective advantages and disadvantages? Any good technical paper, or lecture notes would be appreciated as well.
1
vote
0answers
17 views

Difference between Primal, Dual and Kernel Ridge Regression

I would like to basically ask what the title says. What is the difference between Primal, Dual and Kernel Ridge Regression? People are using all three, and because of the different notation that ...
1
vote
0answers
39 views

Ridge regression in Matlab produces different results than direct calculation [duplicate]

I am trying to run Ridge regression in Matlab (ridge): $L$ is some matrix, $x$ is some random vector and $y=Lx+\alpha n$ is another vector. I expect ...
1
vote
0answers
16 views

How to calculate regularization parameter in generalized ridge regression given the degrees of freedom and input matrix?

I read a Q/A from here which is extremely nice. In the above Q/Answer the tuning parameter $λ$ was a scalar. But in my problem it is a vector $\lambda$ for a generalized ridge regression case. The ...
2
votes
1answer
119 views

Gaussian Process Kernel and Ridge Regression

Can a Dual Ridge Regression produce the same prediction results as a Gaussian Process with a polynomial kernel $K(x,x')=(x^Tx'+1)^2$ in less time complexity (GP is $O(n^3)$ ) using Cholesky ...
0
votes
0answers
31 views

What can be a cause of a extremely high standard coefficient?

I am using RapidMiner to perform linear regression with ridge parameter 1$\text{E}$ -8, min tolerance 0.05 and M5 prime for feature elimination. The std coefficient ...
3
votes
1answer
61 views

Equation 3.49 from Elements of Statistical Learning

I was on page 66 of ESL. I don't know how equation 3.49 on that page is derived. Where does the $N$ in the denominator come from ? Can someone kindly show me the steps in between the lines of equation ...
0
votes
0answers
32 views

On distance between parameters in Ridge regression [duplicate]

In ridge regression we know that as an estimate of $\beta$ and this gives the minimum sum of squares of the residuals: And we know that The question is how to demonstrate that
1
vote
0answers
31 views

Why does not ridge regression perform feature selection although it makes use of regularization?

Could somebody explain why ridge regression does not perform feature selection although it makes use of regularization? So, it penalizes the regression coefficients like LASSO does, but how come we ...
0
votes
0answers
58 views

Estimating Bias in a Linear Regression Model

Is there any way to estimate the bias of the estimate of the betas in a linear regression model when the actual beta values are unknown? The well known Mean Square Error (MSE) criterion is used to ...
1
vote
1answer
117 views

Fastest way to run ridge regression on large datasets where n>>p

Provided that you don't want to do any variable selection: Is there any software which is faster than glmnet at vanilla ridge regression for large datasets?
1
vote
2answers
71 views

Linear regression for time-series prediction

Say we have $N$ time series $X_t^i$ for $i=1...N$and we want to predict a separate time series $Y_t$. Let's consider the following model: $Y_t = \sum_{i} \beta_i X_{t-1}^i $ I am just trying to ...
2
votes
1answer
114 views

Applying ridge regression for an underdetermined system of equations?

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$ can be written as \begin{equation} \begin{array} &\operatorname{min}\ \| y - ...
2
votes
0answers
123 views

Confusion with Vowpal Wabbit's multiple-pass behavior when performing ridge-regression

I have encountered many peculiarities/misunderstandings of Vowpal Wabbit when trying to do online multiple-pass learning. Specifically, I need to solve a Ridge Linear regression problem, with ...
0
votes
0answers
34 views

Linearized Ridge Regression Estimator Under the Mean Squared Error Criterion in a Linear Regression Model

I have been implementing this paper: http://www.tandfonline.com/doi/abs/10.1080/03610918.2011.575506#.UrijOLS_UhI I can send a copy should anybody require it. I think that my implementation is ...
0
votes
0answers
28 views

How to estimate parameter for Biased Regression Estimator

Al-Gereari 2012 introduces the Modified Marquardt Estimator (MME) as: $\hat{\beta^{(r)}(m,k,J)}=T_r[\Delta_r(k)]^{-1}T_r'+m T_{p-r}[\Delta_{p-r}(k)]^{-1}T_{p-r}')(X'Y+kJ)$ where ...
1
vote
1answer
81 views

Coefficients from a penalized Cox PH

I'm using the R package Penalized (0.9-42) on a Cox PH model. I'm using L2 (Ridge) on the grounds that I don't want to shrink my coefficients to 0. I don't understand why when I ask for: ...
0
votes
1answer
84 views

Cross Validation for Ridge Regression

I'm using ridge regression for calculating optimal weights of a set of scores. These scores are correlated so the usage of ridge regression is used for penalizing large values of weights. So the ...
0
votes
2answers
104 views

Ridge Regression in R where coefficients are penalized toward numbers other than zero

Is it possible to penalize coefficients toward a number other than zero in a ridge regression in R? For example, let's say I have dependent variable Y and independent variables X1,X2,X3, and X4. ...
0
votes
0answers
33 views

Can't replicate the results of Golub's Generlized Cross Validation procedure (GCV)

I'm trying to understand this paper: Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter Author(s): Gene H. Golub, Michael Heath and Grace Wahba Source: Technometrics, Vol. ...
1
vote
0answers
59 views

Ridge regression not appropriate for collinearity caused by mathematical constraints on the data

In this paper: Use of the Bootstrap and Cross-Validation in Ridge Regression Author(s): Nancy Jo Delaney and Sangit Chatterjee Source: Journal of Business & Economic Statistics, Vol. 4, No. 2 ...
0
votes
0answers
26 views

Correct degrees of Freedom for Variance estimator in Ridge Regression

When I run a ridge regression on the following data using the Hoerl, Kennard and Baldwin estimator I only get the correct value of k when I use (n-p) where n is the number of observations and p the ...
1
vote
0answers
110 views

Pros and Cons of different types of regression? [closed]

I'm currently studying Linear Regression and am wondering if somebody would help me out with a high-level comparison of different methods. Just to list some: Lasso, Ridge, Elastic Net, Principal ...
0
votes
1answer
89 views

Ridge regression results different in using lm.ridge and glmnet

I applied some data to find the best variables solution of regression model using ridge regression in R. I have used lm.ridge and ...
3
votes
1answer
102 views

Large scale ridge regression

I'm trying to solve a problem of the form $\min_x \frac{1}{2}||Ax-b||^2_2 + \frac{\rho}{2}||x-z||^2_F$ where both $x$ and $b$ are high dimensional, and $b$ is much higher dimensional than $x$. The ...
2
votes
1answer
41 views

Simulating new x's in regression simulation study

One of my homework problems is a simulation that compares three estimators (least squares, ridge regression with known parameters, and ridge regression with estimated parameters) for the following ...
3
votes
0answers
225 views

Estimators for linear regression when multicollinearity is present

I have a multicollinearity problem is a linear regression model and ridge regression was suggested as a solution. So I have spent quite some time researching different ridge regressors in the ...
0
votes
0answers
29 views

Kernel/Basis function design with regularizer

I am solving this problem: $$ \sum_i \parallel f(x_i)- y_i\parallel_2^2 + \lambda <\psi f, \psi f>_{L_2}^2 $$ where the second part $<\psi f, \psi f>_2^2$ is regularizer using the linear ...
1
vote
0answers
111 views

Iterative method to find Ridge Regression Parameter

I have seen a method whereby instead of trying to estimate the ridge parameter (k) directly from the data (using one of the many many ridge parameter estimators in the literature) you solve for it ...
1
vote
0answers
40 views

Degrees of Freedom for Ridge regression without knowing the Ridge Parameter?

There is a very nice post here that gives a neat solution to the problem of finding the ridge parameter when the degrees of freedom are known: How to calculate regularization parameter in ridge ...
5
votes
0answers
83 views

Selecting optimal set of eigenvectors for Principal Components Regression

I am testing various techniques for dealing with strong multi-collinearity (MC) in a regression problem. There have been various comparison papers written between competing techniques such as Ridge ...
0
votes
1answer
92 views

How Can I use some variables selected by LASSO?

I am very new about statistics. So, please understand if my question is somewhat awkward, and please give me related any advice. I have some data set. X = 500 x 100 (500 observations x 100 ...
0
votes
1answer
117 views

How big are regularization parameters values?

I wanted to know how big are the regularization parameter values for ridge or lasso. I have seen most of the places generally using values like 0.1 or 0.01 but in some of my experiments the cross ...
1
vote
1answer
115 views

Issues with ridge regression

I am running ridge regression on some data using a series of regularization params from 0.0001 to 1000. I was amazed to see that high values of lambda param 500-1000 is giving better results with ...
1
vote
1answer
265 views

Derivation of the equation of ridge regression

I am having some issues with the derivation of the equation of ridge regression I know the part without the regularization param like $\beta = (X^TX)^{-1}X^Ty$ But after adding the L2 term ...
1
vote
2answers
147 views

Issues when using neural network

I am having an issue with using neural networks. I started with something simple. I just used nntool with one hidden layer (with one neuron) with linear activation function. For the output also, I ...
2
votes
2answers
108 views

Should I include interactions in ridge regression?

I'm running the a ridge regression broadly analogous to the one at http://www.mathworks.com.au/help/stats/ridge.html. However, I have a lot more predictors (12) than the example (which only has 3). ...
1
vote
1answer
108 views

Implementing linear regression with standardization

I have this confusion related to implementing linear regression with normalization. Let's say I have a training set trainX and ...
0
votes
2answers
230 views

Issues with using glmnet package for matlab

I am trying to use glmnet-matlab pacakge for training my elastic net model on some huge data. My features are of size 13200 and I have around 6000 samples of these. I directly tried to use lassoglm in ...
1
vote
1answer
89 views

Equivalence between Elastic Net formulations

According to Hastie's paper, the elastic net has two equivalent formulations: $\hat{\beta} = \underset{\beta}{\operatorname{argmin}} \left\{ \sum_{i=1}^N(y_i-\sum_{j=1}^p x_{ij} \beta_j)^2 + ...
4
votes
3answers
166 views

Why does regularization of coefficient magnitude improve the generalization of linear regression?

What is the basic argument upon which ridge and lasso regression are based on? I went through Tikhonov regularization wiki where it was mentioned that In many cases, tikhonov matrix is chosen as ...
0
votes
1answer
124 views

Survival analysis with ridge regression in R give same results with different random seeds

I am doing survival analysis using ridge regression. I'm using this R command: coxph(Surv(time, status) ~ ridge(x1, x2, x3), data=DATA) As far as I know, ...
1
vote
1answer
147 views

Stepwise versus L2 regularized logistic regression: dataset-specific performance

I have two data sets from different collections. The second data set is smaller. They were both analyzed with the same methods in order to derive feature sets of 10-30 features each. Each feature set ...
2
votes
0answers
40 views

How to select model when different models are preferred with different seeds?

I am trying to apply the lasso or ridge regression to my data set for the feature selection, but different random seeds produce different models. What is a good or universal way to obtain the final ...
1
vote
0answers
71 views

Prediction Intervals in Ridge Regression?

For some reason, I can't seem to find the formula for prediction intervals in ridge regression anywhere. I know that the coefficient estimates are biased, but are the predictions (dependent variable) ...
1
vote
0answers
142 views

Use of Signal to Noise Ratio to compare Ridge Regression Estimators

On p170 of Gruber's "Regression Estimators" 2010, he shows that the Liu estimator has a smaller bias than the ordinary ridge regressor but also a larger variance. So when the signal to noise ratio is ...
1
vote
0answers
130 views

Information Geometry for Ridge Regression

A question has already been asked about the usefulness of informational geometry. Using information geometry to define distances and volumes…useful? I am reading the book "regression estimators" by ...
0
votes
1answer
111 views

When to use ridge estimator / naive Bayes

I used the Logistic function in weka, to predict a binary class. I have used SimpleLogistic before, but Logistic also seem to give me good results. I did want to clarify if I understand some things ...