Questions tagged [ridge-regression]

A regularization method for regression models that shrinks coefficients towards zero.

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

How to derive the closed-form expression for a ridge regression estimator with an extra vector in the penalty term?

where $\beta_0$ is a known fixed vector in $R^p$. I understand the derivation for the closed-form expression of the ridge regression estimator: But I cannot find anything about when the penalty term ...
2
votes
1answer
26 views

Ridge fit is not orthogonal to ridge residuals

So I'm reading https://arxiv.org/pdf/1509.09169.pdf on ridge regression. On page 8 under Example 1.3 it says From the figure it is obvious that for any $\lambda >0$ the ‘ridge fit’ $\widehat{Y}(\...
2
votes
1answer
24 views

Logistic regression - regularized model always predict lower probabilities on average compared to unregularized model

I have a model that is using L2 regularization. The non-regularized model has a few coefficients with a high positive value, but otherwise the features have very similar coefficients. In the ...
0
votes
0answers
12 views

MSE as a function of the penalty: How to deal with multiple Minima?

The figure below shows the Test-MSE against $\lambda $, the penalty term. There are two minima, one very close to 0 and the other at around 7. These are made-up data I wanted to use in an ...
1
vote
0answers
17 views

Ridge Regression in R: MASS implementation vs User Defined returning different coefficients

I am currently having trouble with the results of a user defined ridge regression function (one that I have created) against that of the MASS::lm.ridge() function. Below is what is defined as my data: ...
1
vote
1answer
49 views

Regularised linear regression with Newton's method?

I am trying to use the Newton's method $\theta^{(t+1)} = \theta^{(t)} - [H^{(t)}]^{-1} [\nabla L(\theta^{(t)})]$ to minimise the following loss fucntion $L(\theta) = (y - X\theta)^T(y-X\theta) + \...
0
votes
0answers
10 views

Which model to use? (cross validation with early stopping)

In this example, to keep things simple we use only 1 training and validation set, and we are trying to find the best regularization parameter for ridge regression. The square loss below is on the ...
2
votes
0answers
40 views

What is the intuition of a dual?

I have been hearing that the Ridge regression is the dual to the GP (Gaussian process regression). What does this mean? Can someone please give an intuition on what 'dual' is. My impression of the '...
0
votes
1answer
73 views

Ridge and Quadratic Programming for Portfolio Norm Optimization

Much like this post: Quadratic Programming and Lasso, I'm trying to integrate RIDGE Penalty in a dedicated quadratic solver. In my case, I am working with quadprog from MATLAB. Unlike LASSO where you ...
2
votes
1answer
48 views

Ridge regression not working for very simple dataset (yet exact same code works on another dataset)

I found some R code for performing ridge regression on the BostonHousing dataset. I tried to use the exact same code on some simple artifical data but it fails and I get the error message: ...
0
votes
0answers
13 views

Equal variance along left and right singular vectors?

Please confirm or reject my line of reasoning: Given SVD of $X$: $X_{NxP}=U_{NxP}D_{PxP}V_{PxP}'$, Variance along ith column vector of $U$ is given by $||X'u_i||^2=u'_iXX'u_i=u'_id_i^2u_i=d_i^2$, ...
3
votes
1answer
45 views

Eigenvalues in Ridge regression [duplicate]

The ridge regression estimate is given by $$\beta^{*}=(X'X+kI)^{-1}X'y, k≥0,$$ where $X$ is the feature matrix. The original paper, Hoerl and Kennard's Ridge Regression: Biased Estimation for ...
0
votes
1answer
21 views

Multiplying a predictor by a constant in Lasso/Ridge regression

If we multiply one of predictors by a constant $c$ in the regression set-up for all data points. What happens to the weights (or specifically weight corresponding to that predictor) if we are doing ...
2
votes
1answer
40 views

Ridge regularization - intuition behind $\lambda$

I have seen many similar questions and I understand that $\lambda$ is some kind of a tuning parameter that decides how much we want to penalize the flexibility of our model. In other words $\lambda$ ...
3
votes
1answer
42 views

Details about Ridge regression [duplicate]

I have a question about the mathematical details of Ridge Regression and I have not been able to find a detailed explanation. For what I know the ridge regression is a penalty term that is used to ...
0
votes
0answers
14 views

Normalize parameter in sklearn Ridge, Lasso, ElasticNet [duplicate]

Is there any risk or disadvantage to set normalize=True when using ridge, lasso or elasticnet or does it only have benefits? And what is the impact on the range of alpha if it is set to True, does it ...
0
votes
0answers
30 views

Box-Cox formation with model selection, regularization, etc

As my data is not normally distributed, I performed the Box-Cox Transformation on the response. ...
0
votes
0answers
17 views

Which one (ridge or lasso) focuses more on the weights that higher?

From my understanding lasso more aggressive bring the weights to zero when the weights are less than 1. While, ridge will more aggressive bring weights to 0 (I know that ridge won't actually bring ...
1
vote
0answers
26 views

Kernel ridge regression and Gaussian Process Regression

One knows that through the both methods mentioned in the title, in regression setting, with the same kernel $K$, the result is the same. It may be a very naive question but why? To me, they are quite ...
5
votes
4answers
304 views

Mean squared error of OLS smaller than Ridge?

I am comparing the mean squared error (MSE) from a standard OLS regression with the MSE from a ridge regression. I find the OLS-MSE to be smaller than the ridge-MSE. I doubt that this is correct. Can ...
1
vote
1answer
90 views

Is this the correct way to run an adaptive LASSO?

I have been using the code here to run an adaptive LASSO in R using glmnet. Essentially it first runs ridge regression to get coefficients for each predictor. It ...
0
votes
1answer
33 views

Angle between $\hat{y}$ and $y$ stays the same as $\lambda$ in ridge regression is adjusted

I was given a thought experiment a while back to think about, but I haven't been able to come up with a solution. The question is For some dataset $X$ with response $Y$, you apply ridge regression. ...
0
votes
0answers
18 views

What is the column space for ridge regression?

In OLS, our $\hat{y}$ lies in the subspace spanned by the columns of $X \in \mathbb{R}^{n \times p}$. For ridge regression, what subspace does $\hat{y}$ lie in? What spans this subspace? I know that $$...
0
votes
1answer
30 views

If there any benefit to using ridge regression in a simple linear regression problem where the aim is prediction?

Consider the following situation: We have a simple linear regression model (as opposed to a multiple regression model or a polynomial regression model). We are interested in prediction rather than ...
0
votes
0answers
33 views

Geometric interpretation of ridge regression

Given a ridge regression variant function where $y \in \mathbf{R^n}, s \in \mathbf{R^d}, X \in \mathbf{R^{dxn}}$ and $C$ a regularization parameter $\in \mathbf{R}$ $w_{crr}= \frac{1}{2}||w||^2 +C||y -...
0
votes
1answer
41 views

Is there any need for regularization in an overdetermined multiple regression problerm?

Supposed I have a small number of features, say 4 or 5, and I have hundreds of data points. That is, I am in an over-determined situation. Is there any benefit to using regularization in this setting ...
3
votes
1answer
62 views

Ridge coefficient estimates do not match OLS estimates when $\lambda$ = 0

I'm trying to understand why ridge regression coefficient estimates (through the glmnet package in R) do not match the ordinary least squares (OLS) estimates in the ...
0
votes
0answers
26 views

Optimal GCV Ridge Regression in Closed Form

I’ve searched all over but I can’t find anywhere a closed form solution to the optimal penalty term in ridge regression using generalized cross validation as the objective function. I’m starting to ...
1
vote
1answer
54 views

Why can't do ridge regression with one predictor?

I'm trying to fit a ridge regression model with a single predictor. However, when I try to do so in three different R packages I get the three following errors: ...
2
votes
1answer
69 views

Why isn't simulation showing that ridge regression better than linear model

I am learning about ridge regression. I was under the impression that ridge regression is valuable because it provides better out of sample predictive accuracy than standard linear models. For example,...
0
votes
0answers
21 views

How does Ridge regression / Regularization help in selecting less or more important features? [duplicate]

Can someone please explain how regularization helps to shrink the " less important " features to zero ? As far as I know , Regularization only penalizes the weights of ALL the features to ...
2
votes
1answer
41 views

What happens to the coefficients of Ridge and Lasso when you have perfect multicollinnearity?

So let's say we ran a Ridge or Lasso regression on $Y \sim X$, and get coefficient $\beta_X$. Now if we duplicate the $X$, and call it $Z$, and then run the same regression on: $Y \sim X + Z$. How ...
0
votes
1answer
39 views

the meaning and metrics of beta coefficients in glmnet Ridge regression

I have done a ridge regression using the 'glmnet' function in R. Then, after finding the optimal lambda parameter, I checked what are the predictors' beta coefficients by extracting glmnet.fit$beta ...
1
vote
0answers
63 views

What does it mean for ridge regression solution to not be “equivariant” under scaling of inputs?

For Ridge Regression, I've seen the statement that the solutions aren't equivariant to scaling of the inputs so we typically preprocess the response and regressors by standardizing. What does "...
0
votes
0answers
18 views

Does Ridge Regression generally improve the condition number?

While learning about Ridge regression and its applications I found a test question about impact of Ridge regression on the condition number. As far as I understand, Ridge regression can decrease the ...
0
votes
0answers
21 views

Linear regression overfitting and regulization

When creating a linear model with many variables, there can be overfitting. Let's say that the training error is 10, and the testing error is 12. So one can use Ridge or Lasso regression to used the ...
0
votes
0answers
39 views

Do residuals sum to zero for ridge regression?

In OLS, residuals are guaranteed to sum to zero because that's how OLS is essentially defined/derived since the residual vector is orthogonal to the column space spanned by by the $p$ independent ...
3
votes
1answer
69 views

What does it mean to have a “gaussian prior?”

When reading up on ridge regression, I saw it stated that it has a "gaussian prior." I realized that I don't know what the word prior means in this context and what it is applied to? I ...
0
votes
0answers
23 views

Penalizing non-OLS models

Let’s consider some common linear time series models for which OLS does not usually yield unbiased coefficient estimates. These include ARIMA and ARIMAX models, regression models with ARIMA errors, ...
6
votes
2answers
161 views

Bayesian interpretation of logistic ridge regression

Most textbooks (also this blog) cover the fact that ridge regression, $$ \hat y = \hat \beta X; \\ \hat \beta = \underset{\beta}{\text{argmin}}\ \ \frac{(y-\beta X)^T(y-\beta X)}{\sigma^2} + \lambda \...
1
vote
1answer
304 views

How to implement the closed form solution of Ridge Regression in Python when intercept is not 0 (fit_intercept=True) without using sklearn?

The well-known closed-form solution of Ridge regression is: I am trying to implement the closed-form using NumPy and then compare it with sklearn. I can get the same result when there is no ...
1
vote
0answers
29 views

Mixed-effects models with lasso penalty and ridge penalty in R [closed]

I am using the PISA 2015 data and trying to run a mixed-effects ridge and lasso regression model Schools will be included as a random effect, and student-level (e.g. motivation, socioeconomic status, ...
4
votes
0answers
37 views

Do variable-selection methods (e.g. Elastic Net; Lasso) invalidate theory-based models in fields where little is known?

I'm caught in a bind about the relationship between theoretical models about how the world works and statistical methods for accurately predicting an outcome in fields where little is known. I ...
1
vote
1answer
30 views

standardizing the MSE

I conducted a Ridge regression with k-fold validation. All the predictors were scaled prior to the regression procedure. To report on the accuracy of my model's prediction, I calculated the MSE in the ...
2
votes
1answer
23 views

Misclassification for test and training sets

I have a problem where I need to provide the misclassification error for both training- and test-set. I am working with logistic regression, so I have a binomial family for my models. I have two ...
0
votes
0answers
28 views

Why Ridge regression doesn't depend on centering $y$ in sklearn?

In sklearn's manual for Ridge they wrote the following about its parameter "fit_intercept": But it seems that Ridge model doesn't depend on whether $y$ is centered or not: ...
0
votes
0answers
33 views

Bias and variance calculation for Ridge estimator of β

I understand how bias and variance for ridge estimator of β are calculated when the model is Y=Xβ + ϵ. But I have the model Y=Xtβ + ϵ. I don't understand if a model like that makes sense, can someone ...
0
votes
0answers
14 views

Ridge regression on one predictor and the reduction of the risk of overfitting

In the book Hands-On Machine Learning with Scikit-Learn and TensorFlow, Chapter 1, the author stated that when doing Ridge Regression (for only one predictor), this regularization reduces the risk of ...
2
votes
1answer
67 views

How does multicollinearity affect the eigenvalues of a matrix?

I have been looking into ridge regression as a method to address multicollinearity in data. I am aware that multicollinearity can cause high variance in coefficient estimates. I have seen equations ...
0
votes
1answer
37 views

In Bayesian Ridge Regression, is the ridge regression parameter $\lambda$ treated as random?

I am having a hard time understanding the Bayesian Ridge Regression. In Bayesian Ridge Regression, is the ridge regression parameter $\lambda$ treated as random? or do we treat the $\lambda$ as given?...

1
2 3 4 5
13