Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

RSS of ridge regression in terms of OLS estimator

In the work by Hoerl, Arthur E., and Robert W. Kennard , "Ridge regression: Biased estimation for nonorthogonal problems." the following formula (3.1) is presented: $$ RSS=(Y-XB)'(Y-XB)=(Y-X\...
0
votes
0answers
13 views

Why does the value of the penalized ridge is divided by 2 in GLMNET? [duplicate]

If you look at GLMNET Vignette, it shows that it solves for the gaussian case: But why does it divide the value of $\parallel \beta \parallel_2^2$ by 2?
0
votes
0answers
9 views

Correlation Effect on Bias in Shrinkage Regression

I want to compare the effect of multicollinearity on the bias as well as variance of predictions via OLS, Ridge and Lasso. I simulated data 100 times. The data consists of 12 $x$ variables and one $y$ ...
1
vote
0answers
26 views

Relationship between Bias/Variance and Covariates in Ridge/Lasso Regression

Suppose I add irrelevant (i.e. no explanatory power) regressors to a ridge/lasso regression. Does this impact the model bias/variance? In the case of OLS, the model bias remains unchanged, while the ...
1
vote
1answer
24 views

Intersection of MSE-Loss and Regularization Term

In several questions [1,2] the graphical intuition of the L1/L2 regularization has been discussed. But, for example in [1], it has been stated that: The solution to the constrained optimization lies ...
1
vote
2answers
38 views

Lasso Regression's role in shrinking the coefficient to zero and Ridge Regression's in not doing the same

How Lasso regression helps feature selection of model by making the coefficient zero? I could see few below with below diagram. Can anyone explain in simple terms how to correlate below diagram with i....
3
votes
2answers
44 views

How to prove ridge estimator residuals variance

The ridge residuals are defined as $\epsilon(\lambda)=y-X\beta^{ridge}(\lambda)$, for the model $y_i=x_i^T\beta+e_i$, where $e_i\sim N(0,\sigma^2)$, and $\beta$ is estimated by the ridge regression ...
11
votes
2answers
693 views

Proving Ridge Regression is strictly convex

Definition of ridge regression $$min_\beta||y-X\beta||_2^2+\lambda||\beta||_2^2, \lambda\ge0$$ you can prove a function is strictly convex if the 2nd derivative is strictly greater than 0 thus But ...
1
vote
0answers
36 views

What are examples of linear learners other than OLS and ridge?

Given $y \in \mathbb{R}^{n}$ and $X \in \mathbb{R}^{n \times p}$, we can define a linear operator $H: \mathbb{R}^n \rightarrow \mathbb{R}^n$ as a mapping from $y$ to the fitted values $\hat{y} \in \...
2
votes
1answer
55 views

Ridge fit is not orthogonal to ridge residuals

So I'm reading https://arxiv.org/pdf/1509.09169.pdf on ridge regression. On page 8 under Example 1.3 it says From the figure it is obvious that for any $\lambda >0$ the ‘ridge fit’ $\widehat{Y}(\...
2
votes
1answer
26 views

Logistic regression - regularized model always predict lower probabilities on average compared to unregularized model

I have a model that is using L2 regularization. The non-regularized model has a few coefficients with a high positive value, but otherwise the features have very similar coefficients. In the ...
0
votes
0answers
17 views

MSE as a function of the penalty: How to deal with multiple Minima?

The figure below shows the Test-MSE against $\lambda $, the penalty term. There are two minima, one very close to 0 and the other at around 7. These are made-up data I wanted to use in an ...
1
vote
0answers
18 views

Ridge Regression in R: MASS implementation vs User Defined returning different coefficients

I am currently having trouble with the results of a user defined ridge regression function (one that I have created) against that of the MASS::lm.ridge() function. Below is what is defined as my data: ...
1
vote
1answer
72 views

Regularised linear regression with Newton's method?

I am trying to use the Newton's method $\theta^{(t+1)} = \theta^{(t)} - [H^{(t)}]^{-1} [\nabla L(\theta^{(t)})]$ to minimise the following loss fucntion $L(\theta) = (y - X\theta)^T(y-X\theta) + \...
0
votes
0answers
11 views

Which model to use? (cross validation with early stopping)

In this example, to keep things simple we use only 1 training and validation set, and we are trying to find the best regularization parameter for ridge regression. The square loss below is on the ...
2
votes
0answers
41 views

What is the intuition of a dual?

I have been hearing that the Ridge regression is the dual to the GP (Gaussian process regression). What does this mean? Can someone please give an intuition on what 'dual' is. My impression of the '...
0
votes
1answer
78 views

Ridge and Quadratic Programming for Portfolio Norm Optimization

Much like this post: Quadratic Programming and Lasso, I'm trying to integrate RIDGE Penalty in a dedicated quadratic solver. In my case, I am working with quadprog from MATLAB. Unlike LASSO where you ...
2
votes
1answer
50 views

Ridge regression not working for very simple dataset (yet exact same code works on another dataset)

I found some R code for performing ridge regression on the BostonHousing dataset. I tried to use the exact same code on some simple artifical data but it fails and I get the error message: ...
0
votes
0answers
20 views

Equal variance along left and right singular vectors?

Please confirm or reject my line of reasoning: Given SVD of $X$: $X_{NxP}=U_{NxP}D_{PxP}V_{PxP}'$, Variance along ith column vector of $U$ is given by $||X'u_i||^2=u'_iXX'u_i=u'_id_i^2u_i=d_i^2$, ...
3
votes
1answer
51 views

Eigenvalues in Ridge regression [duplicate]

The ridge regression estimate is given by $$\beta^{*}=(X'X+kI)^{-1}X'y, k≥0,$$ where $X$ is the feature matrix. The original paper, Hoerl and Kennard's Ridge Regression: Biased Estimation for ...
0
votes
1answer
31 views

Multiplying a predictor by a constant in Lasso/Ridge regression

If we multiply one of predictors by a constant $c$ in the regression set-up for all data points. What happens to the weights (or specifically weight corresponding to that predictor) if we are doing ...
2
votes
1answer
42 views

Ridge regularization - intuition behind $\lambda$

I have seen many similar questions and I understand that $\lambda$ is some kind of a tuning parameter that decides how much we want to penalize the flexibility of our model. In other words $\lambda$ ...
3
votes
1answer
43 views

Details about Ridge regression [duplicate]

I have a question about the mathematical details of Ridge Regression and I have not been able to find a detailed explanation. For what I know the ridge regression is a penalty term that is used to ...
0
votes
0answers
16 views

Normalize parameter in sklearn Ridge, Lasso, ElasticNet [duplicate]

Is there any risk or disadvantage to set normalize=True when using ridge, lasso or elasticnet or does it only have benefits? And what is the impact on the range of alpha if it is set to True, does it ...
0
votes
0answers
33 views

Box-Cox formation with model selection, regularization, etc

As my data is not normally distributed, I performed the Box-Cox Transformation on the response. ...
0
votes
0answers
19 views

Which one (ridge or lasso) focuses more on the weights that higher?

From my understanding lasso more aggressive bring the weights to zero when the weights are less than 1. While, ridge will more aggressive bring weights to 0 (I know that ridge won't actually bring ...
1
vote
0answers
27 views

Kernel ridge regression and Gaussian Process Regression

One knows that through the both methods mentioned in the title, in regression setting, with the same kernel $K$, the result is the same. It may be a very naive question but why? To me, they are quite ...
6
votes
5answers
468 views

Mean squared error of OLS smaller than Ridge?

I am comparing the mean squared error (MSE) from a standard OLS regression with the MSE from a ridge regression. I find the OLS-MSE to be smaller than the ridge-MSE. I doubt that this is correct. Can ...
1
vote
1answer
167 views

Is this the correct way to run an adaptive LASSO?

I have been using the code here to run an adaptive LASSO in R using glmnet. Essentially it first runs ridge regression to get coefficients for each predictor. It ...
0
votes
1answer
33 views

Angle between $\hat{y}$ and $y$ stays the same as $\lambda$ in ridge regression is adjusted

I was given a thought experiment a while back to think about, but I haven't been able to come up with a solution. The question is For some dataset $X$ with response $Y$, you apply ridge regression. ...
0
votes
0answers
18 views

What is the column space for ridge regression?

In OLS, our $\hat{y}$ lies in the subspace spanned by the columns of $X \in \mathbb{R}^{n \times p}$. For ridge regression, what subspace does $\hat{y}$ lie in? What spans this subspace? I know that $$...
0
votes
1answer
48 views

If there any benefit to using ridge regression in a simple linear regression problem where the aim is prediction?

Consider the following situation: We have a simple linear regression model (as opposed to a multiple regression model or a polynomial regression model). We are interested in prediction rather than ...
0
votes
0answers
35 views

Geometric interpretation of ridge regression

Given a ridge regression variant function where $y \in \mathbf{R^n}, s \in \mathbf{R^d}, X \in \mathbf{R^{dxn}}$ and $C$ a regularization parameter $\in \mathbf{R}$ $w_{crr}= \frac{1}{2}||w||^2 +C||y -...
0
votes
1answer
41 views

Is there any need for regularization in an overdetermined multiple regression problerm?

Supposed I have a small number of features, say 4 or 5, and I have hundreds of data points. That is, I am in an over-determined situation. Is there any benefit to using regularization in this setting ...
4
votes
1answer
69 views

Ridge coefficient estimates do not match OLS estimates when $\lambda$ = 0

I'm trying to understand why ridge regression coefficient estimates (through the glmnet package in R) do not match the ordinary least squares (OLS) estimates in the ...
0
votes
0answers
32 views

Optimal GCV Ridge Regression in Closed Form

I’ve searched all over but I can’t find anywhere a closed form solution to the optimal penalty term in ridge regression using generalized cross validation as the objective function. I’m starting to ...
1
vote
1answer
65 views

Why can't do ridge regression with one predictor?

I'm trying to fit a ridge regression model with a single predictor. However, when I try to do so in three different R packages I get the three following errors: ...
2
votes
1answer
72 views

Why isn't simulation showing that ridge regression better than linear model

I am learning about ridge regression. I was under the impression that ridge regression is valuable because it provides better out of sample predictive accuracy than standard linear models. For example,...
0
votes
0answers
21 views

How does Ridge regression / Regularization help in selecting less or more important features? [duplicate]

Can someone please explain how regularization helps to shrink the " less important " features to zero ? As far as I know , Regularization only penalizes the weights of ALL the features to ...
2
votes
1answer
43 views

What happens to the coefficients of Ridge and Lasso when you have perfect multicollinnearity?

So let's say we ran a Ridge or Lasso regression on $Y \sim X$, and get coefficient $\beta_X$. Now if we duplicate the $X$, and call it $Z$, and then run the same regression on: $Y \sim X + Z$. How ...
0
votes
1answer
53 views

the meaning and metrics of beta coefficients in glmnet Ridge regression

I have done a ridge regression using the 'glmnet' function in R. Then, after finding the optimal lambda parameter, I checked what are the predictors' beta coefficients by extracting glmnet.fit$beta ...
1
vote
0answers
80 views

What does it mean for ridge regression solution to not be “equivariant” under scaling of inputs?

For Ridge Regression, I've seen the statement that the solutions aren't equivariant to scaling of the inputs so we typically preprocess the response and regressors by standardizing. What does "...
0
votes
0answers
18 views

Does Ridge Regression generally improve the condition number?

While learning about Ridge regression and its applications I found a test question about impact of Ridge regression on the condition number. As far as I understand, Ridge regression can decrease the ...
0
votes
0answers
22 views

Linear regression overfitting and regulization

When creating a linear model with many variables, there can be overfitting. Let's say that the training error is 10, and the testing error is 12. So one can use Ridge or Lasso regression to used the ...
0
votes
0answers
48 views

Do residuals sum to zero for ridge regression?

In OLS, residuals are guaranteed to sum to zero because that's how OLS is essentially defined/derived since the residual vector is orthogonal to the column space spanned by by the $p$ independent ...
3
votes
1answer
90 views

What does it mean to have a “gaussian prior?”

When reading up on ridge regression, I saw it stated that it has a "gaussian prior." I realized that I don't know what the word prior means in this context and what it is applied to? I ...
0
votes
0answers
26 views

Penalizing non-OLS models

Let’s consider some common linear time series models for which OLS does not usually yield unbiased coefficient estimates. These include ARIMA and ARIMAX models, regression models with ARIMA errors, ...
6
votes
2answers
188 views

Bayesian interpretation of logistic ridge regression

Most textbooks (also this blog) cover the fact that ridge regression, $$ \hat y = \hat \beta X; \\ \hat \beta = \underset{\beta}{\text{argmin}}\ \ \frac{(y-\beta X)^T(y-\beta X)}{\sigma^2} + \lambda \...
1
vote
1answer
414 views

How to implement the closed form solution of Ridge Regression in Python when intercept is not 0 (fit_intercept=True) without using sklearn?

The well-known closed-form solution of Ridge regression is: I am trying to implement the closed-form using NumPy and then compare it with sklearn. I can get the same result when there is no ...
1
vote
0answers
42 views

Mixed-effects models with lasso penalty and ridge penalty in R [closed]

I am using the PISA 2015 data and trying to run a mixed-effects ridge and lasso regression model Schools will be included as a random effect, and student-level (e.g. motivation, socioeconomic status, ...

1
2 3 4 5
13