Questions tagged [ridge-regression]
A regularization method for regression models that shrinks coefficients towards zero.
602
questions
1
vote
0answers
10 views
How to derive the closed-form expression for a ridge regression estimator with an extra vector in the penalty term?
where $\beta_0$ is a known fixed vector in $R^p$. I understand the derivation for the closed-form expression of the ridge regression estimator:
But I cannot find anything about when the penalty term ...
2
votes
1answer
26 views
Ridge fit is not orthogonal to ridge residuals
So I'm reading https://arxiv.org/pdf/1509.09169.pdf on ridge regression. On page 8 under Example 1.3 it says
From the figure it is obvious that for any $\lambda >0$ the ‘ridge fit’ $\widehat{Y}(\...
2
votes
1answer
24 views
Logistic regression - regularized model always predict lower probabilities on average compared to unregularized model
I have a model that is using L2 regularization. The non-regularized model has a few coefficients with a high positive value, but otherwise the features have very similar coefficients. In the ...
0
votes
0answers
12 views
MSE as a function of the penalty: How to deal with multiple Minima?
The figure below shows the Test-MSE against $\lambda $, the penalty term. There are two minima, one very close to 0 and the other at around 7.
These are made-up data I wanted to use in an ...
1
vote
0answers
17 views
Ridge Regression in R: MASS implementation vs User Defined returning different coefficients
I am currently having trouble with the results of a user defined ridge regression function (one that I have created) against that of the MASS::lm.ridge() function. Below is what is defined as my data:
...
1
vote
1answer
49 views
Regularised linear regression with Newton's method?
I am trying to use the Newton's method
$\theta^{(t+1)} = \theta^{(t)} - [H^{(t)}]^{-1} [\nabla L(\theta^{(t)})]$ to minimise the following loss fucntion
$L(\theta) = (y - X\theta)^T(y-X\theta) + \...
0
votes
0answers
10 views
Which model to use? (cross validation with early stopping)
In this example, to keep things simple we use only 1 training and validation set, and we are trying to find the best regularization parameter for ridge regression. The square loss below is on the ...
2
votes
0answers
40 views
What is the intuition of a dual?
I have been hearing that the Ridge regression is the dual to the GP (Gaussian process regression). What does this mean? Can someone please give an intuition on what 'dual' is.
My impression of the '...
0
votes
1answer
73 views
Ridge and Quadratic Programming for Portfolio Norm Optimization
Much like this post: Quadratic Programming and Lasso, I'm trying to integrate RIDGE Penalty in a dedicated quadratic solver. In my case, I am working with quadprog from MATLAB. Unlike LASSO where you ...
2
votes
1answer
48 views
Ridge regression not working for very simple dataset (yet exact same code works on another dataset)
I found some R code for performing ridge regression on the BostonHousing dataset. I tried to use the exact same code on some simple artifical data but it fails and I get the error message: ...
0
votes
0answers
13 views
Equal variance along left and right singular vectors?
Please confirm or reject my line of reasoning:
Given SVD of $X$: $X_{NxP}=U_{NxP}D_{PxP}V_{PxP}'$,
Variance along ith column vector of $U$ is given by $||X'u_i||^2=u'_iXX'u_i=u'_id_i^2u_i=d_i^2$, ...
3
votes
1answer
45 views
Eigenvalues in Ridge regression [duplicate]
The ridge regression estimate is given by
$$\beta^{*}=(X'X+kI)^{-1}X'y, k≥0,$$
where $X$ is the feature matrix. The original paper, Hoerl and Kennard's Ridge Regression: Biased Estimation for ...
0
votes
1answer
21 views
Multiplying a predictor by a constant in Lasso/Ridge regression
If we multiply one of predictors by a constant $c$ in the regression set-up for all data points. What happens to the weights (or specifically weight corresponding to that predictor) if we are doing ...
2
votes
1answer
40 views
Ridge regularization - intuition behind $\lambda$
I have seen many similar questions and I understand that $\lambda$ is some kind of a tuning parameter that decides how much we want to penalize the flexibility of our model. In other words $\lambda$ ...
3
votes
1answer
42 views
Details about Ridge regression [duplicate]
I have a question about the mathematical details of Ridge Regression and I have not been able to find a detailed explanation. For what I know the ridge regression is a penalty term that is used to ...
0
votes
0answers
14 views
Normalize parameter in sklearn Ridge, Lasso, ElasticNet [duplicate]
Is there any risk or disadvantage to set normalize=True when using ridge, lasso or elasticnet or does it only have benefits? And what is the impact on the range of alpha if it is set to True, does it ...
0
votes
0answers
30 views
Box-Cox formation with model selection, regularization, etc
As my data is not normally distributed, I performed the Box-Cox Transformation on the response.
...
0
votes
0answers
17 views
Which one (ridge or lasso) focuses more on the weights that higher?
From my understanding lasso more aggressive bring the weights to zero when the weights are less than 1. While, ridge will more aggressive bring weights to 0 (I know that ridge won't actually bring ...
1
vote
0answers
26 views
Kernel ridge regression and Gaussian Process Regression
One knows that through the both methods mentioned in the title, in regression setting, with the same kernel $K$, the result is the same.
It may be a very naive question but why?
To me, they are quite ...
5
votes
4answers
304 views
Mean squared error of OLS smaller than Ridge?
I am comparing the mean squared error (MSE) from a standard OLS regression with the MSE from a ridge regression. I find the OLS-MSE to be smaller than the ridge-MSE. I doubt that this is correct. Can ...
1
vote
1answer
90 views
Is this the correct way to run an adaptive LASSO?
I have been using the code here to run an adaptive LASSO in R using glmnet. Essentially it first runs ridge regression to get coefficients for each predictor. It ...
0
votes
1answer
33 views
Angle between $\hat{y}$ and $y$ stays the same as $\lambda$ in ridge regression is adjusted
I was given a thought experiment a while back to think about, but I haven't been able to come up with a solution.
The question is
For some dataset $X$ with response $Y$, you apply ridge regression. ...
0
votes
0answers
18 views
What is the column space for ridge regression?
In OLS, our $\hat{y}$ lies in the subspace spanned by the columns of $X \in \mathbb{R}^{n \times p}$.
For ridge regression, what subspace does $\hat{y}$ lie in? What spans this subspace?
I know that
$$...
0
votes
1answer
30 views
If there any benefit to using ridge regression in a simple linear regression problem where the aim is prediction?
Consider the following situation:
We have a simple linear regression model (as opposed to a multiple regression model or a polynomial regression model).
We are interested in prediction rather than ...
0
votes
0answers
33 views
Geometric interpretation of ridge regression
Given a ridge regression variant function where $y \in \mathbf{R^n}, s \in \mathbf{R^d}, X \in \mathbf{R^{dxn}}$ and $C$ a regularization parameter $\in \mathbf{R}$
$w_{crr}= \frac{1}{2}||w||^2 +C||y -...
0
votes
1answer
41 views
Is there any need for regularization in an overdetermined multiple regression problerm?
Supposed I have a small number of features, say 4 or 5, and I have hundreds of data points. That is, I am in an over-determined situation.
Is there any benefit to using regularization in this setting ...
3
votes
1answer
62 views
Ridge coefficient estimates do not match OLS estimates when $\lambda$ = 0
I'm trying to understand why ridge regression coefficient estimates (through the glmnet package in R) do not match the ordinary least squares (OLS) estimates in the ...
0
votes
0answers
26 views
Optimal GCV Ridge Regression in Closed Form
I’ve searched all over but I can’t find anywhere a closed form solution to the optimal penalty term in ridge regression using generalized cross validation as the objective function. I’m starting to ...
1
vote
1answer
54 views
Why can't do ridge regression with one predictor?
I'm trying to fit a ridge regression model with a single predictor. However, when I try to do so in three different R packages I get the three following errors:
...
2
votes
1answer
69 views
Why isn't simulation showing that ridge regression better than linear model
I am learning about ridge regression. I was under the impression that ridge regression is valuable because it provides better out of sample predictive accuracy than standard linear models. For example,...
0
votes
0answers
21 views
How does Ridge regression / Regularization help in selecting less or more important features? [duplicate]
Can someone please explain how regularization helps to shrink the " less important " features to zero ? As far as I know , Regularization only penalizes the weights of ALL the features to ...
2
votes
1answer
41 views
What happens to the coefficients of Ridge and Lasso when you have perfect multicollinnearity?
So let's say we ran a Ridge or Lasso regression on $Y \sim X$, and get coefficient $\beta_X$. Now if we duplicate the $X$, and call it $Z$, and then run the same regression on: $Y \sim X + Z$. How ...
0
votes
1answer
39 views
the meaning and metrics of beta coefficients in glmnet Ridge regression
I have done a ridge regression using the 'glmnet' function in R. Then, after finding the optimal lambda parameter, I checked what are the predictors' beta coefficients by extracting glmnet.fit$beta ...
1
vote
0answers
63 views
What does it mean for ridge regression solution to not be “equivariant” under scaling of inputs?
For Ridge Regression, I've seen the statement that the solutions aren't equivariant to scaling of the inputs so we typically preprocess the response and regressors by standardizing.
What does "...
0
votes
0answers
18 views
Does Ridge Regression generally improve the condition number?
While learning about Ridge regression and its applications I found a test question about impact of Ridge regression on the condition number. As far as I understand, Ridge regression can decrease the ...
0
votes
0answers
21 views
Linear regression overfitting and regulization
When creating a linear model with many variables, there can be overfitting. Let's say that the training error is 10, and the testing error is 12.
So one can use Ridge or Lasso regression to used the ...
0
votes
0answers
39 views
Do residuals sum to zero for ridge regression?
In OLS, residuals are guaranteed to sum to zero because that's how OLS is essentially defined/derived since the residual vector is orthogonal to the column space spanned by by the $p$ independent ...
3
votes
1answer
69 views
What does it mean to have a “gaussian prior?”
When reading up on ridge regression, I saw it stated that it has a "gaussian prior." I realized that I don't know what the word prior means in this context and what it is applied to?
I ...
0
votes
0answers
23 views
Penalizing non-OLS models
Let’s consider some common linear time series models for which OLS does not usually yield unbiased coefficient estimates. These include ARIMA and ARIMAX models, regression models with ARIMA errors, ...
6
votes
2answers
161 views
Bayesian interpretation of logistic ridge regression
Most textbooks (also this blog) cover the fact that ridge regression,
$$
\hat y = \hat \beta X;
\\
\hat \beta = \underset{\beta}{\text{argmin}}\ \ \frac{(y-\beta X)^T(y-\beta X)}{\sigma^2}
+ \lambda \...
1
vote
1answer
304 views
How to implement the closed form solution of Ridge Regression in Python when intercept is not 0 (fit_intercept=True) without using sklearn?
The well-known closed-form solution of Ridge regression is:
I am trying to implement the closed-form using NumPy and then compare it with sklearn. I can get the same result when there is no ...
1
vote
0answers
29 views
Mixed-effects models with lasso penalty and ridge penalty in R [closed]
I am using the PISA 2015 data and trying to run a mixed-effects ridge and lasso regression model Schools will be included as a random effect, and student-level (e.g. motivation, socioeconomic status, ...
4
votes
0answers
37 views
Do variable-selection methods (e.g. Elastic Net; Lasso) invalidate theory-based models in fields where little is known?
I'm caught in a bind about the relationship between theoretical models about how the world works and statistical methods for accurately predicting an outcome in fields where little is known. I ...
1
vote
1answer
30 views
standardizing the MSE
I conducted a Ridge regression with k-fold validation. All the predictors were scaled prior to the regression procedure.
To report on the accuracy of my model's prediction, I calculated the MSE in the ...
2
votes
1answer
23 views
Misclassification for test and training sets
I have a problem where I need to provide the misclassification error for both training- and test-set. I am working with logistic regression, so I have a binomial family for my models. I have two ...
0
votes
0answers
28 views
Why Ridge regression doesn't depend on centering $y$ in sklearn?
In sklearn's manual for Ridge they wrote the following about its parameter "fit_intercept":
But it seems that Ridge model doesn't depend on whether $y$ is centered or not:
...
0
votes
0answers
33 views
Bias and variance calculation for Ridge estimator of β
I understand how bias and variance for ridge estimator of β are calculated when the model is Y=Xβ + ϵ. But I have the model Y=Xtβ + ϵ. I don't understand if a model like that makes sense, can someone ...
0
votes
0answers
14 views
Ridge regression on one predictor and the reduction of the risk of overfitting
In the book Hands-On Machine Learning with Scikit-Learn and TensorFlow, Chapter 1, the author stated that when doing Ridge Regression (for only one predictor), this regularization reduces the risk of ...
2
votes
1answer
67 views
How does multicollinearity affect the eigenvalues of a matrix?
I have been looking into ridge regression as a method to address multicollinearity in data.
I am aware that multicollinearity can cause high variance in coefficient estimates. I have seen equations ...
0
votes
1answer
37 views
In Bayesian Ridge Regression, is the ridge regression parameter $\lambda$ treated as random?
I am having a hard time understanding the Bayesian Ridge Regression.
In Bayesian Ridge Regression, is the ridge regression parameter $\lambda$ treated as random? or do we treat the $\lambda$ as given?...
